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Symposium Overview 

The goal of the symposium is to present findings from five separate investigations 
undertaken as part of the Texas College and Career Readiness Initiative (TCCRI), a 
statewide legislatively mandated set of activities designed to increase the number of 
students attending college and succeeding once they reach college. This symposium 
highlights the dimension of the TCCRI that is focused on improving alignment between 
high school and college by defining and operationalizing postsecondary expectations for 
college readiness. These activities were undertaken to define what constitutes 
readiness, validate the definition against current practice in entry-level college courses, 
develop “reference courses” to serve as benchmarks for college expectations, explore 
the relationship between college readiness in key cross-disciplinary skills and the 
expectations present in two-year certificate courses, and analyze the degree to which 
placement tests measure the knowledge and skills defined as college ready. The 
session will consider findings from these five aspects of the Initiative within a broader 
policy context in order to illustrate significance both at the level of each study 
individually and for the Initiative collectively. 

Texas has been a leader in the movement to align high-school curriculum and 
assessments with college readiness expectations, beginning with the initiation of 
Closing the Gaps, a plan aimed to ensure high school students in Texas were prepared 
for college and the workforce (THECB, 2009). In May 2006, the Texas Legislature 
passed House Bill 1 , a major piece of legislation that included multiple initiatives related 
to high school success and college readiness. Its primary goal is to increase the number 
of Texas high school students who graduate ready to succeed in college and 21st 
Century careers. 

The TCCRI represents a significant advancement in the field of systems alignment for 
college readiness. No other state has undertaken such a comprehensive approach to 
identifying, validating, and implementing the knowledge and skills necessary for college 
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success. The Initiative consists of four phases that address a range of issues and 
policies designed to improve statewide alignment between secondary and 
postsecondary systems. Findings from activities undertaken in Phases l-lll of the TCCRI 
are presented in this symposium. 

Conceptual Framework for the Symposium 

Preparing all students for college and career readiness is a growing national priority as 
increasing numbers of parents, educators, business leaders, and politicians emphasize 
the importance of a highly educated workforce and citizenry (Achieve, 2006). Reaching 
this difficult goal is complicated by a number of factors. These include the separate 
governance systems for K-12 and postsecondary education (Conley, 2003), the lack of 
shared information and expectations across educational systems, the limitations of the 
existing means for determining college eligibility (Conley, 2007), the changes in the 
demographics of American public school students (National Center for Educational 
Statistics, 2007), and even the well-intentioned efforts of states to institute exit 
examinations (Brown & Conley, 2007). 

In order to prepare students to succeed in the U.S. economy and society, states have 
adopted a strategy of aligning educational expectations vertically and holding schools 
accountable for achieving defined outcomes (Susan H. Fuhrman, 1993; S.H. Fuhrman & 

Elmore, 2004). For essentially all states, this means alignment of standards and 
assessments across elementary, middle, and high school. The key missing link is 
alignment of expectations between secondary and postsecondary education (Conley, 

2005). As a concept, alignment is the underlying driver behind a range of state and 
federal policies heretofore confined to K-12 education and only now being considered 
seriously in relation to the transition from high school to college. 

The symposium capitalizes on the systemic efforts by Texas to align postsecondary and 
secondary education, using it as a framework to present a series of findings from an 
interrelated set of activities designed to set the stage for more concrete actions to align 
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high school and college. The intent is to balance each presentation between an 
overview of each study and its findings and a discussion of the overall implications for 
policy and practice. The studies included in this symposium include: 

1 . Development of the Texas College and Career Readiness Standards (CCRS) 

2. Validation of the Texas College and Career Readiness Standards Against 
Current practice in Entry-Level College Courses 

3. Creation of “reference courses” in each of 20 entry-level college course subject 
area 

4. Analysis of alignment between the CCRS cross-disciplinary standards and the 
expectations present in entry-level courses leading to two-year certificates at 
public postsecondary institutions in Texas 

5. Analysis of alignment between the CCRS and placement tests commonly used in 
Texas postsecondary institutions 

Scientific or Scholarly Significance of the Studies 

The findings from these studies are important for a number of reasons, most important 
among them is how they illustrate a state’s attempt to improve high school-college 
alignment at the systems level and how such an effort can serve to inform other states. 

Given current interest in “common core standards,” by the US Department of Education 
and most states, a presentation that highlights postsecondary efforts to set college- 
ready standards and to align entry-level expectations to such standards will be highly 
germane. If and when common core standards are adopted by states, the stage will be 
set for discussions about the alignment of such standards with postsecondary 
readiness. Having an example of one state’s efforts to anticipate the need to align high 
school and college expectations will be useful and important. 

Beyond the general findings from the five presentations, the symposium serves as a 
sort of proof of concept for the notion of state-based initiatives to align high school and 
college. The papers offer insights into the process and methodological issues 
associated with an initiative issue of this sort and also highlight important relationships 
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between college readiness standards and other aspects of the postsecondary system, 
such as entry-level courses and placement tests. These insights can help inform policy 
leaders nationally as they begin to grapple with alignment issues in earnest in the wake 
of national efforts to focus high school standards on college and career readiness. 

Finally, the results shed needed light on the methodological issues associated with 
alignment work of this nature, which is becoming increasingly widespread and higher 
stakes. Understanding how to go about establishing the content and expectations of 
entry-level college courses will be an important area of investigation, and the 
methodological strategies to do so will be of increasing interest. This symposium will 
showcase a range of strategies and techniques for working in this area. 
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Project #1 : 

Development of the Texas College and Career Readiness 
Standards (CCRS) 

Overview 

In May 2006, the 79 th Texas Legislature (Third Called Session) passed House Bill 1 , a 
major piece of legislation that included multiple initiatives related to high school success 
and college and career readiness. This legislation added Section 28.008, entitled 
“Advancement of College Readiness in Curriculum,” to Chapter 28 of the Texas 
Education Code. Its goal was to increase the number of students who graduate from 
Texas high schools ready to succeed in college and 21 st Century careers. 

In response to elements of this legislation, the Educational Policy Improvement Center 
(EPIC) was awarded a contract issued by the Texas Higher Education Coordinating 
Board (THECB) to facilitate the development and implementation of college and career 
readiness standards as part of the Texas College and Career Readiness Initiative 
(TCCRI). The purpose of the TCCRI is to develop and implement college and career 
readiness standards and activities to improve alignment between secondary and 
postsecondary education, resulting in an increased number of students prepared for 
college and career success. 

The TCCRI represents a significant advancement in the field of college and career 
readiness. No other state has undertaken such a comprehensive approach to 
identifying, validating, and implementing the knowledge and skills necessary for college 
and career success. 

In 2007, EPIC facilitated the development of the College and Career Readiness 
Standards (CCRS), in partnership with the THECB and the Texas Education Agency 
(TEA). Vertical teams composed of secondary and postsecondary instructors 
representing all regions of the state engaged in the development process. These 
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standards were adopted by the THECB in January 2008, and were sent to the 
Commissioner of Education and State Board of Education for incorporation into the 
Texas Essential Knowledge and Skills (TEKS) in April 2008. For details, please see: 

Texas College and Career Readiness Standards, available online at: 
http://www.thecb.state.tx.us/collegereadiness/TCRS.cfm. 

The purpose of this central aspect of the TCCRI was to develop a set of standards that 
represent what students must know and be able to do to succeed in entry-level college 
courses in English/language arts, mathematics, science, and social studies. In addition 
to content specification by subject area, they contain a stand-alone section that 
specifies cross-disciplinary skills. These skills include many cognitive strategies and 
techniques necessary for success in a wide range of college courses. 

Methodology 

The method used to generate the standards was a form of criterion-based expert 
judgment that combined convergent consensus with theme analysis. Expert teams 
comprised of 39 carefully selected and vetted secondary and postsecondary faculty 
members met four times over a period of eight months, reviewed relevant extant 
standards in their subject area and then completed online exercises between meetings. 

The postsecondary faculty all taught entry-level courses. The secondary educators all 
had expertise or experience developing content standards. Drafts were reviewed by 
national experts and were made available for public comment. Results from the reviews 
were incorporated into subsequent drafts. 

The teams were instructed to use as their reference point the knowledge and skills a 
student would need to be ready to succeed in a credit-bearing entry-level general 
education college course at a state postsecondary institution. “Succeed” was defined as 
being able to complete the entry-level course at a level that would allow further study in 
the subject area, if the student chose to do so. 
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Between meetings, EPIC analysts conducted theme analysis on input gathered at each 
meeting and developed online draft versions that participants reviewed and commented 
upon between meetings. In this fashion, numerous successive versions came to 
approximate better the consensus view of the participants. 

Results 

The final version specified, by subject, the major standards, enduring understandings, 
key concepts, performance expectations, and performance indicators necessary for 
college success in each of the subject areas, including English/language arts, 
mathematics, science, and social studies. In addition, a separate section, the cross- 
disciplinary standards, detailed the key cognitive skills and foundational skills necessary 
for college and career readiness that transcend subject matter. The four nested levels 
are organized as follows: 

I. Key Content - Keystone ideas of a discipline that reverberate as themes 
throughout the curriculum (designated by Roman numerals). Example: I. Numeric 
Reasoning 

A. Organizing Component - Knowledge and subject areas that organize a 
discipline around what students should retain, be able to transfer, and apply to 
new knowledge and skills (designated by capital letters). Example: A. Number 
representation 

1. Performance Expectation - Knowledge and skills that represent 
important ideas of the current understanding of each organizing concept as 
well as the multiple contexts in which each organizing concept can be 
manifest (designated by numbers). Example: 1 . Compare real numbers. 

a. Performance Indicator - examples of how to assess and measure 
performance expectations. This is not intended to be an exhaustive or 
prescriptive list. The operating premise is that the more of these or other 
similar indicators a student is successfully able to demonstrate, the greater 
the probability that the student will be prepared to succeed in college 
(designated by lowercase letters). Example: a. Classify numbers as 
natural, whole, integers, rational, irrational, real, imaginary, and/or 

complex. 
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The resulting standards were made available for public review, and Texas 
postsecondary faculty in particular were encouraged to review and comment upon the 
draft. Overall, almost 1 ,000 individuals submitted comments. Table 1 lists the number of 
participants submitting comments by subject area. 



Table 1: Participation in Public Comment Process by Subject Area 



Subject 


Number of Individuals Submitting Comments 


English 


271 


Math 


197 


Science 


173 


Social Studies 


165 


Cross-Disciplinary 


165 


Total 


971 



The comments were then analyzed and reviewed by the VT members as they prepared 
their final drafts. The final version of the standards was submitted to the Commissioner 
of Higher Education, who presented them to the THECB for adoption at its January 
2008 meeting. The CCRS were approved unanimously (the THECB adopted the first 
three levels; the fourth level, Performance Indicators, are intended to serve only as 
examples) and sent to the Commissioner of Education and the State Board of Education 
for incorporation into the Texas Essential Knowledge and Skills (TEKS). 

Significance 

The Texas College and Career Readiness Standards are significant for several reasons. 
The method used to develop the standards moved beyond the traditional professional 
judgment approach by triangulating current entry-level college and secondary faculty 
members’ knowledge and expertise using an iterative process comprised of in-person 
and online components, incorporating public comments, and validating the standards 
against practices in entry-level college courses to confirm they do represent the 
knowledge and skills necessary for postsecondary success. In addition, the VT 
members used research and standards developed over the past 20 years in the relevant 
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subject areas, using them as the foundation for creating the CCRS. 

Whereas the CCRS serve as an important advancement in standards development, 
they do differ fundamentally from traditional state high school graduation standards. 

Overall, they have two different reference points. High school graduation standards tend 
to focus on graduation (obtaining a high school diploma) as the terminal reference point. 

They focus more on content knowledge and are the basis for state testing or 
accountability requirements. The CCRS do not specify what students must master to 
graduate from high school; rather, they set out what students need to know in order to 
have a reasonable probability of success in their introductory college course at two- and 
four-year institutions. The operating premise is that the more of these standards a 
student can successfully demonstrate, the higher the probability for student success in 
entry-level coursework and the avoidance of remedial placements. In addition, the two 
sets of standards can complement each other: most state standards systems do not 
continue to twelfth grand currently, ending at tenth or sometimes eleventh grade, 
making the alignment to college readiness standards a reasonable exit level high school 
extension. 

The CCRS also contain a significant distinguishing feature and an important dimension 
of college and career readiness: the identification and inclusion of the cross-disciplinary 
standards. The necessity for including the cross-disciplinary standards emerged as the 
Vertical Teams began to identify important knowledge and skills that transcended their 
particular subject areas. For example, the Science and Mathematics Vertical Teams 
collaborated to ensure that the standards in both areas addressed the mathematical 
skills necessary to be successful in entry-level college sciences courses. Through these 
cross-team conversations, the need for the cross-disciplinary standards became 
apparent. These standards represent elements of learning that cut across disciplines, 
and are organized into two areas: Key Cognitive Skills and Foundational Skills. The Key 
Cognitive Skills specify intellectual behaviors that are prevalent in entry-level college 
courses (such as intellectual curiosity, reasoning, problem solving, academic behaviors, 
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work habits, and academic integrity). Foundational Skills consist of proficiencies 
students need to be able to transfer knowledge and apply it across the curriculum (such 
as reading, writing, conducting research, understanding and using data, and using 
technology). One of the strongest findings from the subsequent validity studies of the 
OCRS (reported throughout the remainder of the this symposium) is the high level of 
alignment of the cross-disciplinary standards have with faculty expectations for 
readiness in entry-level college courses in both general education and Career and 
Technical Education courses. Through the identification and validation of the high level 
of importance for success in entry-level college work, the cross-disciplinary skills 
represent an important body of knowledge all educators are responsible for teaching 
within their own discipline and context. 

The CCRS also represent a unique state-level effort to connect a state’s K-12 standards 
with college and career readiness. The state’s willingness to undertake significant 
legislatively-sponsored statewide alignment activities could serve as a model for other 
states. These standards serve as the basis for multiple alignment activities. They can be 
compared to the state high school standards and assessments as a means to 
determine how well the two are aligned. In addition, districts and high schools can use 
the standards to align their course expectations more directly with college readiness. 

This is particularly important for courses beyond the level of or content not covered in 
the state high school examinations that currently have no explicit reference points. 

Overall, Texas is an important state in its ability to influence textbook publishers, which 
could lead to the standards influencing curriculum and textbooks in other states. The 
CCRS also serve as an important reference point for the higher education system, 
enabling additional alignment activities such as departmental internal and external 
course alignment self-examinations, and explorations of placement tests. 
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Project #2: 

Validation of the Texas College and Career Readiness 
Standards (CCRS) Against Current practice in Entry-Level 
College Courses 

Overview 

The College and Career Readiness Standards (CCRS) were then validated against 
current practice by analyzing the content of over 930 syllabi from 20 core entry-level 
general education courses taught at two- and four-year postsecondary institutions 
throughout the state. The complete findings from this research are available in the 
report Examining the Alignment between the Texas College and Career Readiness 
Standards and Entry-Level College Courses at Texas Postsecondary Institutions 
(Referred to hereafter as the TCCRI Phase II Validity Study) released by EPIC in 
October 2008. 

During Phase I of this project, completed in January 2008, EPIC facilitated the 
development of the CCRS, in partnership with the THECB and the Texas Education 
Agency (TEA). Vertical teams composed of secondary and postsecondary instructors 
representing all regions of the state engaged in the development process. These 
standards were adopted by the THECB in January 2008, and were sent to the 
Commissioner of Education and State Board of Education for incorporation into the 
Texas Essential Knowledge and Skills (TEKS) in April 2008. 

The next step in the process was to conduct an alignment analysis to compare the 
standards to what is actually being taught in entry-level college courses in 
postsecondary institutions throughout Texas. The purpose was to establish the validity 
of the standards as an accurate representation of the key knowledge and skills 
necessary for college and career readiness and success. The results of the study can 
be used to affirm the accuracy of elements of the CCRS and to identify areas where 
additions, deletions, or modifications to the standards should be considered. 
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Methodology 

This study was designed to answer the following question: How do the CCRS compare 
to what is currently taught in entry-level college courses at Texas institutions of higher 
education? This question was addressed through a validity analysis identifying the 
alignment and any gaps between the CCRS and current instructional content and 
practices in 20 entry-level college courses. Over 800 entry-level college course 
instructors in eight subject areas representing 20 separate course titles rated the 
importance of what they taught in those courses in relation to the CCRS. They also 
submitted their course syllabi. External reviewers analyzed the match between the 
instructor ratings of importance and the content of their syllabi to gauge the accuracy of 
the self-ratings. 

To start this review process, the Commissioner of Higher Education asked the Chief 
Academic Officers of all state and private colleges and universities in Texas (with the 
exception of 12 health-related and private career schools) to designate a College 
Readiness Special Advisor (CRSA) at their respective campuses. The THECB invited 
138 public and private institutions of higher education to designate a college readiness 
special advisor; 108 institutions accepted the invitation. Two public and 28 private 
institutions did not respond to the invitation. 

The THECB charged the CRSAs with soliciting course nominations from their respective 
institutions. To select the 20 course titles for which data would be collected, the THECB 
identified courses with high enrollment statewide among entry-level students along with 
“gatekeeper” courses that are prerequisite requirements for different majors. The 
THECB selected courses according to the Texas Common Course Numbering System, 
a uniform set of course designations that the majority of Texas institutions of higher 
education employ to help facilitate the transfer of entry-level courses between 
institutions. The course titles selected include the following: 

• BIOL 1406 - Biology for Science Majors I 
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• BIOL 1408 - Biology for Non-Majors I (lecture & lab) 

• BIOL 2401 - Anatomy and Physiology I 

• CHEM 1405 - Introductory Chemistry I 

• CHEM 1412 - General Chemistry II (lecture & lab) 

• ENGL 1301 - Composition I 

• ENGL 1302 - Composition II 

• ENGL 2332 - World Literature I 

• GOVT 2301 - American Government I 

• GOVT 2302 - American Government II 

• GOVT 2305 - Federal Government 

• GOVT 2306 - Texas Government 

• HIST 1301 - U. S. History I 

• HIST 1302 -U.S. History II 

• MATH 1314 - College Algebra 

• MATH 1324 - Mathematics for Business & Social Sciences I 

• MATH 1342 - Elementary Statistical Methods 

• PHYS 1401 - College Physics I 

• PHYS 1405 - Elementary Physics I 

• PSYC 2301 - General Psychology 



Between October 2007 and January 2008, Course nominations were collected from 
CRSAs via a web-based application designed for this specific purpose. Nominations 
included courses that were considered exemplary, were aligned with the CCRS, or that 
stood out in some other way as the best representation of an entry-level course in the 
course title area. The special advisors nominated 1 ,21 1 courses by submitting the 
faculty member’s name and contact information and the institution-specific course title 
when appropriate. The distribution of nominated courses reflected relatively closely the 
distribution of the state public two-year and four-year institutions by geographic area. 

In March 2008, instructors whose courses had been nominated received an email 
notifying them of the availability of the course submission site. Submitting faculty 
members provided information on course objectives, class size, grading policy, texts 
used, prerequisite and pathways courses, and percent of students who enter their 
course well prepared. Instructors were then instructed to upload an electronic version of 
the course syllabus. Table 2 reports the distribution of all course submissions 
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(completed and partial) by region and institution type. In total, EPIC collected 960 
course submissions, including 913 complete submissions and 47 partial. 



Table 2: Distribution of all Course Submissions by Region and Institution Type 



Region 


Community 

College 


4 Year Public 
University 


Technical 

College 


Private 

College 


Total 


Central 


58 


33 


3 


1 


95 


Gulf Coast 


93 


63 


0 


0 


156 


High Plains 


35 


21 


0 


0 


56 


Metroplex 


126 


54 


0 


0 


180 


Northwest 


44 


16 


3 


0 


63 


South 


87 


69 


14 


1 


171 


Southeast 


11 


29 


31 


0 


71 


Upper East 


58 


16 


4 


6 


84 


Upper Rio Grande 


17 


10 


0 


0 


27 


West 


30 


27 


0 


0 


57 


Total 


559 (58%) 


338 (35%) 


55 (6%) 


8(1%) 


960 



After providing course information and submitting a syllabus, instructors completed an 
online rating form that asked them to answer the following question for each relevant 
content area standard and cross-disciplinary standard: “How necessary is this element 
in preparing students to succeed in my course?” Respondents chose one of five 
options: Most necessary, More necessary, Less necessary, Least necessary, or Not 
necessary. Instructors also had the option of skipping any section of the standards if 
that section was not relevant to their course (e.g., biology instructors could skip the 
physics section). After selecting a response option for each standard, instructors then 
chose one or more rationale statements to explain their reasoning. (See Appendix B for 
a list of scale items and rationale statements.) The rationale statements were designed 
to provide greater clarity to understand the ratings. For example, an instructor might 
designate a standard as Not necessary or Least necessary tor one of several reasons. 
The standard could not be necessary to succeed in the course because it was irrelevant 
to the subject area, or because it would be covered in a subsequent course. Distinctions 
of this nature are important when considering discrepant standards. Including the 
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rationale statements were particularly valuable in interpreting the reasons why specific 
standards were not well aligned. 

External Review to Check the Reliability of Self-Ratings 

A common criticism of self-ratings is the potential lack of reliability of such ratings. Self- 
ratings have the advantage of generating a tremendous amount of information in an 
efficient and cost-effective manner. However, such ratings should be verified so that 
findings based on the ratings can be interpreted with confidence as being representative 
of the actual state of the phenomenon being investigated, in this case the importance of 
each CCRS in preparing students for success in entry-level college courses. 

The method selected to examine the accuracy of the self-ratings was a concurrent 
expert review. This method uses trained subject-matter experts to use course syllabi 
from each submitted course to verify the assertions made by the instructor of that 
course on each CCRS. In other words, if an instructor stated that a particular CCRS was 
Most important for success in a course, the external reviewer would examine the course 
syllabus to determine whether it contained evidence to support the instructor’s self- 
ratings of alignment with the CCRS. The expert reviewer could choose among three 
statements: Evident, Not evident, or Not applicable. By selecting Evident, the expert 
reviewer indicated that the standard was Reasonable to infer, Stated verbatim, or 
Implied in the course documents. 

Expert reviewers examined a sample consisting of approximately 20 percent of courses 
submitted in each course title area. Table 2 presents the percent of instructor responses 
confirmed by external reviewers by course title. The rate of confirmation ranges from 96 
percent in BIOL 2401 (Anatomy and Physiology I) to 50 percent in CHEM 1405 
(Introductory Chemistry I). 
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Table 2: Percent of Instructor Responses Confirmed by Course Title 



Course Title 


# of Courses 
Reviewed 


% of Instructor Responses 
Confirmed 


BIOL 1406 - Biology for Science Majors 1 


11 


89% 


BIOL 1408 - Biology for Non-Majors 1 (lecture & lab) 


6 


86% 


BIOL 2401 - Anatomy and Physiology 1 


9 


96% 


CHEM 1405 - Intro Chemistry 1 


9 


50% 


CHEM 1412 - General Chemistry II (lecture & lab) 


8 


60% 


ENGL 1301 - Composition 1 


13 


90% 


ENGL 1302 - Composition II 


12 


94% 


ENGL 2332 - World Literature 1 


11 


92% 


GOVT 2301* - American Government 1 


8 


89% 


GOVT 2302* - American Government II 


6 


83% 


GOVT 2305* - Federal Government 


4 


77% 


GOVT 2306* - Texas Government 


4 


91% 


HIST 1301 - U. S. History 1 


11 


85% 


HIST 1302- U. S. History II 


11 


79% 


MATH 1314 - College Algebra 


15 


88% 


MATH 1324 - Mathematics for Business & Social Sciences 1 


12 


92% 


MATH 1342 - Elementary Statistical Methods 


11 


87% 


PHYS 1401 - College Physics 1 


9 


63% 


PHYS 1405 - Elementary Physics 1 


4 


68% 


PSYC 2301 - General Psychology 


10 


53% 



Results indicate acceptable to very high matches between instructor self-ratings of 
courses against the CCRS and the expert reviewer confirmation of the ratings. Only five 
courses fell below the 75 percent level of verification. Given the inherent difficulties of 
determining instructor intent from a syllabus and the complexity and number of the 
College and Career Readiness Standards, the levels at which external reviewers found 
evidence of the CCRS within the syllabi chosen for review is sufficiently high to have 
confidence that the instructor self-ratings can be taken as accurate representations of 
course content. 

Determining Alignment Thresholds 

Once all data from instructors were received and analyzed, it was possible to establish 
the criterion point for determining that alignment existed or did not exist between the 
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CCRS and the course content. In essence, this was a standard setting activity in its own 
right. For each individual CCRS statement, the modal instructor response was 
determined. Each standard could have a score of 1 through 5, representing each of the 
response options. The modes for each CCRS statement were examined via a scatter 
plot in order to identify overall patterns. In some cases, a CCRS statement had multiple 
modes. This analysis confirmed the appropriateness of collapsing the categories of 
Most necessary and More necessary, using this new category to identify alignment 
between the CCRS statement and instructor self-rating of CCRS importance. CCRS 
statements for which the modal score was Less necessary were labeled as 
inconsistently aligned, and CCRS statements that had a modal score of Least 
necessary or Not necessary were identified as being not aligned. 

Results 

Ultimately, over 800 instructors representing 87 IHE and four college districts submitted 
over 900 syllabi and ratings from 20 different entry-level, credit-bearing courses. Results 
from the analysis revealed that the CCRS are highly aligned with entry-level college 
courses in Texas. Rates of alignment by subject area for all standards were 99 percent 
in social studies, 97 percent in English/language arts, 87 percent in mathematics, and 
86 percent in science. For the cross-disciplinary standards, 100 percent are aligned 
across the four subject areas (90 percent are aligned within each of the four subject 
areas individually). Whereas all of the CCRS may not be aligned in any single course, 
an examination across all courses within a given subject area reveals the high degree of 
alignment between the CCRS and all entry-level courses in that subject. Table 3 below 
lists the results by subject area, and also includes the overall levels of alignment for the 
cross-disciplinary skills both aggregated and by subject area. 
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Table 3: Summary of Overall Alignment by Subject Area and Cross-disciplinary Skills 



Subject 


Aligned 


Inconsistently 

Aligned 


Not Aligned 


Multimodal 


All English 


97% 


2% 


1% 


0% 


All Math 


87% 


8% 


5% 


0% 


All Science 


86% 


14% 


0% 




All Social Studies 


99% 


1% 


0% 


0% 


Cross-Disciplinary 
All Subjects 


100% 


0% 


0% 


0% 


Cross-Disciplinary 
All English 


98% 


0% 


2% 


0% 


Cross-Disciplinary 
All Math 


91% 


9% 


0% 


0% 


Cross-Disciplinary 
All Science 


100% 


0% 


0% 


0% 


Cross-Disciplinary 
All Social Studies 


98% 


2% 


0% 


0% 



Note: for a detailed exploration of the alignment results at the individual standard level within each subject area and 
course, please refer to the original TCCRI Phase II Validity Study. 



Two notable exceptions to the overall high level of alignment emerged in this study. 
First, three key content areas were found to be not well aligned: Geometric Reasoning 
in mathematics; and Earth and Space Sciences and Environmental Science in science. 
The reason for the lower levels of alignment in these three areas is due to the fact none 
of the entry-level courses selected for inclusion in the data collection for this study 
typically cover that content, and an additional study would be necessary to analyze the 
relationship between those standards and the appropriate entry-level courses not 
included in this study. Therefore, the overall rates of alignment for all standards in 
mathematics and science alignment understate the actual degree of alignment by 
including standards in these three areas not included in the data collection. The 
adjusted rates of alignment for the standards represented in the data collection are 93 
percent for mathematics and 92 percent for science. 

Secondly, there was a notably low level of alignment between the physics CCRS and 
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current practice in entry-level physics courses. Only 56 percent of the 50 physics 
standards were rated most necessary or more necessary for preparation for success in 
entry-level physics courses, compared to 91 percent in biology and 94 percent in 
chemistry. Further analysis of the rationale statements provided by the physics 
instructors found the primary reason that the physics standards were not rated as 
aligned was because the faculty taught those standards as new material at the college 
level. When presented with these findings, the original OCRS Science Vertical Team 
recommended the physics standards to remain unchanged due to the nature of the 
discrepancy. Having the opportunity to learn the physics standards in the secondary 
setting would only help better prepare students for what the postsecondary faculty will 
be teaching in their entry-level physics courses. 

Significance 

This study is important because it is the first large-scale validity study of college 
readiness standards undertaken at a state level. Previous college readiness standards 
were developed by expert judgment panels and were often buttressed by exemplar 
materials. But none of these went so far as to validate the standards against current 
practice in college classrooms by using instructor review and syllabus analysis to 
confirm the expert judgment process. This study offers insight into the methods for 
conducting a validation study of this nature in addition to presenting the empirical 
findings specific to the CCRS. It is important to have validated college readiness 
standards particularly in states that are considering aligning high school and college 
expectations, standards, or assessments. The validated college readiness standards 
can then be used for planning purposes between state K-12 education departments and 
postsecondary system offices. Given the high degree of investment that education 
departments in most states have in the state’s high school standards and assessments, 
it is unlikely most states will make changes in these standards to improve system 
alignment unless it can be proven that the college readiness standards are a valid 
representation of current practice in the state’s entry-level college courses. 
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Project #3: 

Creation of Reference Course Profiles in Entry-Level 
College Courses 

Overview 

A separate component of the Phase II Study analyzed the results of the course ratings 
to identify common practice within current entry-level courses and develop 18 Reference 
Course Profiles. The Reference Course Profiles are composite courses designed to 
represent the content and rigor of what is typically being taught currently in entry-level 
college courses, and provide a snapshot of current practice (not best practice). A profile 
includes a course syllabus with significant additional detail along with attendant course 
materials, such as assignments, assessments, and scoring rubrics. The reference 
courses do not represent best practice; instead, each of the 18 reference courses was 
intended to represent normative practice at a wide range of postsecondary institutions 
within Texas. While in practice any individual course at a postsecondary institution might 
differ from the reference course, enough consistency will exist for the reference course 
to be a useful target for secondary teachers to use in preparing students for college. A 
more detailed description of this research can be found in the last section of the report 
Examining the Alignment between the Texas College and Career Readiness Standards 
and Entry-Level College Courses at Texas Postsecondary Institutions (or the TCCRI 
Phase II Validity Study) released by EPIC in October 2008. 

The purposes of the Reference Courses are two-fold. At the secondary level, instructors 
can refer to the materials as they prepare their students for the course content they will 
encounter when they reach college. At the postsecondary level, the materials serve as a 
point of comparison that faculty can use when creating or refining entry-level courses. 

Whereas the use of the Reference Course Profiles is purely voluntary, the goal for 
institutions of higher education is to ensure that entry-level courses are aligned with the 
CCRS, contain college-level content, and are cognitively challenging. By making 
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expectations more transparent, the Reference Course Profiles will help students, 
educators, and policymakers understand more clearly and reach agreement more 
quickly on the nature of the student preparation necessary for college success. 

Methodology 

This study was designed to answer the following question: What are the most common 
components of and current practices in entry-level college courses in Texas 
representing 18 specific course titles? To answer this question, 18 Reference Course 
Profiles were created through a linked criterion-based analytic process. In short, this 
approach linked several steps including identifying the modal courses and designing the 
profiles. The CCRS and the modal faculty expectations relative to the standards in 
entry-level courses within a course title were used as the criteria throughout the 
process. The modal course profile represents the typical instructor response pattern for 
each course title. Using the modal course profile, 10 individual courses were identified 
as the most highly aligned with the typical aggregate rating results for each course title. 

The course materials from these modal courses paint a picture of the most frequent 
current instructional priorities and practices for the specified courses in Texas 
institutions of higher education. 

The Reference Course Profiles were compiled from data collected from faculty ratings, 
syllabi, and additional course documents submitted by instructors in entry-level college 
courses throughout Texas during the data collection process described in the Phase II 
Study in the previous section. Table 4 shows the distribution of submitted syllabi by 
institution type. This table reports only usable submitted syllabi, and does not reflect the 
number of instructor rating completions or partial submissions. 



Table 4: Distribution of Submitted Syllabi by Institution Type 



Course 


Syllabi Submitted: 
2-year 


Syllabi Submitted: 
4-year 


Syllabi Submitted: 
Total 


BIOL 1406 


26 


33 


59 


BIOL 1408 


16 


17 


33 
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Course 


Syllabi Submitted: 
2-year 


Syllabi Submitted: 
4-year 


Syllabi Submitted: 
Total 


BIOL 2401 


24 


21 


45 


CHEM 1405 


19 


30 


49 


CHEM 1412 


21 


23 


44 


ENGL 1301 


30 


37 


67 


ENGL 1302 


27 


34 


61 


ENGL 2332 


24 


32 


56 


GOVT 2301 


10 


21 


31 


GOVT 2302 


10 


17 


27 


HIST 1301 


19 


31 


50 


HIST 1302 


21 


32 


53 


MATH 1314 


28 


40 


68 


MATH 1324 


29 


30 


59 


MATH 1342 


21 


32 


53 


PHYS 1401 


21 


27 


48 


PHYS 1405 


11 


11 


22 


PSYCH 2301 


24 


30 


54 


Total 


381 


498 


879 



Creation of the Modal Courses 

The first step in the linked criterion-based analytic process was to identify the subset of 
courses most aligned with current practice in entry-level college courses in Texas. The 
data analyzed consisted of a scale item response for each standards statement. Scale 
item responses were represented by the numbers one through six and corresponded 
with Not necessary, Least necessary, Less necessary, More necessary, Most 
necessary, and Skip section, respectively. As in the alignment study, responses left 
blank were treated as missing data and excluded from the analysis. 

Then, a “modal course profile” was created that represented the typical response 
pattern for the set of rated standards for the identified courses in each course title area, 
18 in total. The modal course was determined by the aggregation of the results of the 
instructors’ self-ratings representing the necessity of each CCRS for successful 
preparation for the course. In other words, the modal course profile reflects the CCRS 
as they are most typically represented in that course title area. 



Improving Alignment Between Postsecondary and Secondary Education 



25 



epic 

I Educational Policy 
Improvement Center 

Proximity Measures 

Once the modal course profiles were generated, the next linked step was to calculate a 
measure reflecting the percent agreement between the ratings for each course and the 
modal course ratings (a percent match measure) along with three proximity measures 
that examined the similarity between each of the selected courses in a course title area 
and the modal course in that area. The proximity measures included: 1) a Euclidean 
distance measure, 2) a measure of the correlation of the response pattern for each 
course and the modal course, and 3) the cosine value of the angle between the 
response pattern for each course and the response pattern for the modal course. These 
three proximity measures, though distinct, resulted in course orderings that were highly 
related to one another. 

Next, a factor analysis considered all proximity measures as well as the percent match 
value to compute a final score for each course using a linear combination of the four 
similarity measures. These factor scores were used to rank-order the courses in order of 
similarity to the modal course based on the combination of similarity measures. The ten 
most similar courses for each course title then become the basis for building Reference 
Course Profiles that integrate the practices found across these courses, paying special 
attention to the most similar courses. 

The ranking of these factor scores were used to rank-order a total of 180 courses 
across the 18 course title areas, i.e., the ten courses most aligned to the modal course 
for each of the 18 Reference Course Profiles titles. The distribution of these 180 
courses is similar to the distribution of all courses submitted, although community and 
technical colleges are slightly overrepresented, and four-year universities and private 
colleges are slightly underrepresented. Private colleges are absent from the list of most 
representative courses. 

Additional materials, such as assignments and assessments, were collected from 
instructors of these identified courses early in Phase III, beginning Fall 2008. Design 
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teams created the Reference Course Profiles through a criterion-based professional 
judgment process in which decisions about what to include in the profile are made 
based on alignment with the CCRS and the content of the ten highly aligned courses per 
course title area. 

These additional course artifacts provided deeper insight into the ways in which 
instructor expectations relate to the CCRS. Design teams were formed for each content 
area, and each team was composed of three postsecondary instructors. These design 
teams synthesized the results from the linked criterion-based analytic process to create 
the Reference Course Profiles. The resulting Reference Course Profiles will then serve 
to demonstrate how and why the relevant CCRS for that course title area are important 
to success in the course. 

Results 

Design teams created the 18 Reference Course Profiles through a criterion-based 
professional judgment process. Decisions about what to include in the Reference 
Course Profile were made based on alignment with the CCRS and the course materials 
from the ten modal (highly aligned) courses per course title area. The additional course 
artifacts provided deeper insight into the ways in which instructor expectations relate to 
the CCRS. Design teams were formed for each content area, and each team was 
composed of three postsecondary instructors. These design teams synthesized the 
results from the linked criterion-based analytic process to create the Reference Course 
Profiles. The resulting Reference Course Profiles not only provide a snapshot of current 
practice, but also demonstrate how and why the relevant CCRS for that course title area 
are important to success in the course. 
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Reference Course Profile Template 

This next section outlines the template that was used by each of the design teams to 
develop the Reference Course Profiles. The template includes all of the section 
headings of the profile, and short descriptions. For a complete Reference Course Profile 
example, please refer to “Appendix C: Reference Course Profile: English Composition 
II.” 

Course Name 
Course Number 

Course description appearing in the Lower Division Academic Course Guide Manual. 

Reference Course Profile: Overview and Purpose 

A Reference Course Profile represents current practice in entry-level college courses in 
Texas. Twenty-six course profiles were developed from an extensive study of over 900 
entry-level general education and CTE college courses offered at Texas institutions of 
higher education. Each course profile consists of a course description, sample syllabus, 
reference lists, attendant course materials (such as assignments, assessments, and, in some 
cases, scoring rubrics), and the Texas College and Career Readiness Standards addressed 
within the course. 

The Profiles provide a reference point for college readiness that fosters increased 
transparency between secondary and postsecondary education. They are not intended to 
be mandatory, prescriptive, or best practice. 



Prereq u isites and Prior Knowled g e 

Most college-level syllabi do not list all prior knowledge required to succeed in the 
course; it is usually an underlying assumption. For the purpose of this Reference Course 
Profile, the required prior knowledge and skills students need to be successful in the 
course are explicitly stated to help both secondary and postsecondary faculty establish 
goals and expectations for their students. The knowledge and skills reflected in this section 
are pulled directly from the Texas College and Career Readiness Standards (CCRS), written 
and validated by Texas faculty during the 2007-8 school year. The CCRS are available 
online at: http://www.thecb.state.tx.us/collegereadiness/CRS.pdf 



Prior to enrolling in this course, students must satisfy Texas Success Initiative (TSI) 
requirements set by the institution as described in Coordinating Board rule (Texas 
Administrative Code, Chapter 4, Subchapter C). 

An instructor might also require a pre-semester diagnostic test to help the student 
assess his or her current ability in . 
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In addition, students should have the skills reflected in the following College and Career 
Readiness Standards. Only the specific standards and performance expectations 
pertinent to the course are listed below. 

College and Career Readiness Standards 
I. 

A. 

1 . 

2 . 

Cross-Disciplinary Standards 

I. Key Cognitive Skills 

II. Foundational Skills 

Course Objectives 

Course objectives include the course-specific skills and knowledge that students will 
possess upon completion of the course. They assist postsecondary faculty in clarifying the 
goals of their courses and provide a clear picture of the expectations students will 
encounter once they begin college. This sample list of objectives was adapted from syllabi 
submitted in 2008 by Texas college faculty. 



In this course, [course title], students will learn . 

Upon completion of the course the student will be able to: 

1 . 

2 . 

Sample Textbooks and Materials 

This list is comprised of texts that appear on course syllabi submitted in 2008 by faculty 
teaching entry-level college courses most representative of current practice in Texas. This 
list is not exhaustive, prescriptive, or required. 



Sample Methods of Instruction 

Students should be prepared to encounter a variety of instructional methods, as faculty 
indicate the use of several beyond the lecture format. The list of methods of instruction has 
been adapted from course syllabi submitted in 2008 by faculty teaching entry-level 
college courses most representative of current practice in Texas. The approximate 
percentage of time allocated to each instructional method is also indicated. 
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1 . Lecture - X% 

a. Lecture is defined as . 

b. Lectures will take place in the form of . 

2. Full-class Discussion - X% 

a. Students will be expected to come to class ready to contribute to class 
discussion. 

b. Students will be expected to listen and respond to each other’s comments. 

3. Method of Instruction - X% 

Sam p le Assi g nments 

A typical number of formal assignments for [course title] is . The course will also 

include a number of other types of assignments including . Below are the kinds of 

assignments that might be expected and the percentage of the final grade each might 
carry. 



1. Written Work - X% 

a. Statement re: appropriate format. 

b. Statement re: appropriate citation style. 

c. Statement re: appropriate writing style. 

d. Statement re: length requirement: 

2. Other assignment type - X% 

a. Statement of information necessary to inform student expectations. 

3. Other assignment type - X% 

a. Statement of information necessary to inform student expectations. 

Sample Assessments 

Typically [course title] will include formal examinations. Typically the 

culminating assessment for the course will be . 



Specific Assessments: 

1. Quizzes -X% 

a. Statement of information necessary to inform student expectations. 

2. Final Exam - X% 

a. Statement of information necessary to inform student expectations. 
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Sample Schedule 

Samplings of assignments and assessments have been provided in the Reference Course 
Profile materials. Bolded blue text indicates when a document is provided and a link to 
that document is available. The time allocated for students to complete the assignment is 
also indicated. The list of topics, as well as the overall pacing of the course, has been 
adapted from course syllabi submitted in 2008 by faculty teaching entry-level college 
courses most representative of current practice in Texas. 



Week 


Reading/Discussion Topics 


Assignments & Assessments 


1 




Assignment 1: 


2 




Complete exercise. Take quiz. 


3 




Assignment 2: 


4 




Take quizzes. 


5 




Assignment 3: 

Complete exercise. Take quiz. 


6 




Take quizzes. 


7 




Assignment 4: 

Take quizzes. 


8 




Exam 1 - Midterm Assessment 


9 




Assignment 5: 


10 




Take quiz. 


11 




Assignment 6: 


12 




Complete exercise. Take quiz. 


13 




Assignment 7: 

Class presentations 


14 




Assignment 8: 

Final conferences 


15 


Exam 2 - Final Exam 
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Sam p le Class Policies and Ex p ectations 

Students often enter college unaware of expectations regarding attendance, participation, 
behavior, grading, and academic integrity. Faculty frequently include detailed policies and 
expectations in their syllabi, making explicit to students the standards of successful 
participation. Institutional-level policies are often included as well. 



Attendance Policy 

Students are expected to attend class regularly and to complete assignments on the 
days specified. 

Expectations 

Student Conduct 

• No permitted in the classroom. 

• Disruptive behavior will not be tolerated. 

• Examples of disruptive behavior include . 

Academic Behaviors 

Students are expected to . 

Grading Policy 

Oven/jew of Grades and Grading. Standards 

• A (90-100) = excellent/performance beyond mastery 

• B (80-89) = above average/beyond basic mastery 

• C (70-79) = average mastery 

• D (60-69) = below average 

• F (0-59) = failure 

Percentages ot Grade 

• Assignment Type - X% 

• Assessment Type - X% 

• Quizzes - X% 

Academic Integrity Policy 

Sample List of Student Resources 

The following list is representative of the resources mentioned in 2008 entry-level college 
course syllabi in Texas. The resources listed are indicative of the expectation that 
postsecondary students take responsibility for their own learning. As often explicitly stated 
in course syllabi, students are expected to take advantage of these resources, and may 
even be required to do so. 
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Su pp lementar y Documents 

The section lists the additional documents included within the Reference Course Profile, 
but not included in the Sample Schedule. These supplementary materials are intended to 
enhance both instruction and student learning, and may be used at instructors' discretion. 



Significance 

The current phase of the Texas College and Career Readiness Initiative uses the 
Reference Course Profiles as a primary reference point in the creation of College 
Readiness Assignments that are being designed to assess college readiness. College 
Readiness Assignments, scoring guides, and work samples that align with the 
Reference Course Profiles will be disseminated statewide as part of a professional 
development effort to assist educators in improving student preparation for credit- 
bearing entry-level college coursework and improving transparency across the 
secondary and postsecondary systems. 

The Reference Course Profiles are important tools for systems alignment because they 
create an operational connection between the CCRS and actual practices in college 
courses. They help move the alignment discussion beyond anecdote and assertion 
regarding the nature of postsecondary expectations by offering concrete, specific 
descriptions of college courses against which high school programs of study can be 
aligned. For the first time, a state has sponsored a system wide analysis of entry-level 
college courses, thereby providing a research-based point of reference. 

An additional use of the profiles is as a resource for improvements to postsecondary 
education. One of the results of the decentralization of postsecondary governance and 
instruction is that it is essentially impossible to compare practices across a set of 
courses with the same title offered at different institutions, or even at the same 
institution. The Reference Course Profile is a potential tool for comparing the 
expectations, content coverage, and challenge level present in any individual course to 
a more independent standard. While the profile is not designed to standardize 
instruction across institutions, nor represent best practice, it is a useful resource for 
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helping to enhance consistency across entry-level courses, which can foster improved 
alignment between secondary and postsecondary education, if secondary educators 
have consistent targets toward which to align their courses and instruction. 
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Project #4: 

Alignment Between the CCRS and Expectations in Entry- 
Level Career and Technical Education Courses 

Overview 

In Fall 2008, EPIC was awarded an additional contract issued by the THECB to explore 
the relationship between the cross-disciplinary standards and Career and Technical 
Education (CTE) courses. Under this contract, EPIC replicated the validation 
methodology of the TCCRI Phase II Validity Study, by comparing the CCRS cross- 
disciplinary standards to nine CTE courses at two-year institutions throughout the state. 

This report summarizes the results from the study. The complete findings from this 
research are available in the report Texas College and Career Readiness Initiative: 

Texas Career and Technical Education Phase I Alignment Analysis Report { referred to 
hereafter as the Phase I CTE Study) released by EPIC in March, 2009. 

The purpose of this study was to examine the relationship between the Texas College 
and Career Readiness Standards (CCRS) and what is currently being taught in nine 
entry-level Career and Technical Education (CTE) courses throughout Texas. The 
findings from this research will enable high school faculty to determine the degree to 
which what they are teaching is aligned with the knowledge and skills necessary for 
success in a community college or technical college environment. Furthermore, both 
high school and postsecondary faculty teaching CTE courses will have a concrete 
benchmark against which they can compare the challenge levels of their courses. 

A unique feature of the CCRS is the inclusion of cross-disciplinary skills that span all 
subject areas. These are the foundational cognitive skills that underlie and connect all 
disciplinary areas and that students need to be able to apply across a variety of 
contexts and subject matter. They relate to 21 st Century learning and work environments 
in which the cross-disciplinary skills are prerequisites to solving many of the most 
important problems students will encounter in college and the workplace. The cross- 
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disciplinary standards are divided into two areas: Key Cognitive Skills, such as 
reasoning, problem solving and conducting research; and Foundational Skills to process 
and create content knowledge, such as reading, writing, and data analysis. 

The Phase I CTE Study consists of two phases designed to identify the relationship 
between CTE courses and the cross-disciplinary CCRS, and then to illustrate those 
relationships in more concrete fashion. The two phases are outlined below: 

Phase I: Alignment Analysis 

Nine entry-level CTE courses were analyzed against the CCRS cross-disciplinary 
standards, which are arranged in four nested levels. The THECB adopted the 
first three levels; the fourth level includes Performance Indicators intended to 
serve only as examples. This study analyzed the first three levels, which 
represent the Key Cognitive and Foundational Skills and knowledge necessary 
for college and career preparation. The cross-disciplinary skills are organized as 
follows: 

I. Key Cognitive Skills - overarching skills that are necessary for 
success across the curriculum. Example: I. Key Cognitive Skills 

A. Organizing Component - cognitive skills for which a range of 
specific strategies can be developed and mastered. Example: A. 
Intellectual Curiosity 

1. Performance Expectation - specific strategies that make 
up the cognitive skill. Example: 1 . Engage in scholarly inquiry 
and dialogue. 

a. Performance Indicator - examples of how to 
assess and measure performance expectations. This 
is not intended to be an exhaustive list. Example: a. 
Identify what is known, not known, and what one 
wants to know in a problem. 
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Phase II: Reference Course Profile Development 

Based on the course data collected and analyzed in Phase I, design teams made 
up of postsecondary CTE content experts developed composite Reference 
Course Profiles to represent the content and rigor of courses in which entering 
college students should be prepared to succeed. 

Methodology 

This Phase I study was designed to answer the following question: 

How do the standards contained in the cross-disciplinary skills section of the 
Texas College and Career Readiness Standards compare to what is currently 
taught in a range of entry-level Career and Technical Education course areas at 
Texas institutions of higher education? 

This question was addressed by replicating the research design from the TCCRI Phase 
II Validity Study analyzing the relationship between the CCRS and entry-level general 
education courses. The validation methodology included: working with institutional 
liaisons to nominate CTE instructors to participate in the study, developing an online 
document collection and self-ratings tool, collecting course documents (including syllabi, 
assignments, assessments, and scoring rubrics) and self-ratings of the level of 
necessity of each cross-disciplinary standard for preparation for the course, providing 
quality control and technical assistance, and using the results to analyze and report on 
the level of alignment between the cross-disciplinary standards and the CTE courses. 

The key differences between the TCCRI Phase II Validity Study and this study are 
twofold. First, the TCCRI Phase II Validity Study examined entry-level general education 
courses in English/Language Arts (ELA), mathematics, science, and social studies 
around which the content standards of the CCRS were organized. This study examines 
a representative range of CTE courses comprising subject matter that varies 
significantly from the content standards of the CCRS. 

Second, the TCCRI Phase II Validity Study examined the relationship between specific 
subject area standards and the related entry-level courses (for example, the ELA 
standards were analyzed against entry-level composition and literature courses). That 
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report also compared all courses in the core subject areas to the cross-disciplinary 
standards. The CTE study compared representative courses only to the cross- 
disciplinary standards. The content standards included in the CCRS were designed to 
represent the knowledge and skills necessary for success in the related entry-level 
courses, not the universe of entry-level courses. The cross-disciplinary skills, on the 
other hand, were those standards that were found to transcend subject matter, and are 
therefore, can be used as a framework for examining all CTE courses. 

Data Collection and Analysis Methods 

Between November 2008 and January 2009, 136 entry-level CTE course instructors at 
43 different postsecondary institutions throughout Texas representing 157 courses in 
seven subject areas and nine separate courses rated the importance of each cross- 
disciplinary standard in relation to their course(s). Instructors teaching entry-level CTE 
college courses submitted data through a web-based application. 

The data collection process began with the College Readiness Special Advisors at 51 
two-year public postsecondary institutions soliciting course nominations from their 
respective institutions. The THECB identified course titles that enrolled significant 
numbers of students statewide among entry-level CTE students. The THECB selected 
courses according to the Texas Common Course Numbering System, a uniform set of 
course designations that the majority of Texas institutions of higher education employ to 
help facilitate the transfer of entry-level courses between institutions. The course titles 
selected consist of the following: 

• ACNT 1303 Introduction to Accounting I 

• BMGT 1303 Principles of Management 1 



1 1ncludes courses with course title MG NT 1303 Principles of Management 



Improving Alignment Between Postsecondary and Secondary Education 



38 



epic 

I Educational Policy 
Improvement Center 



• DFTG 1309 Basic CAD 

• DFTG 1405 Technical Drafting 

• ITSC 1301 Introduction to Computers 

• ITSC 1401 Introduction to Computers 

• MRKG 131 1 Principles of Marketing 

• POFI 1301 Computer Applications I 

• POFT 1301 Business English 

Course nominations were collected from the Special Advisors between October 2008 
and mid-January 2009. The Special Advisors nominated 21 1 courses by submitting the 
faculty member’s name and contact information and the institution-specific course title 
when it was known. 

In December 2008, instructors whose courses had been nominated received an email 
asking them to log in to the online course-submission site. The online course 
submission process included the following five steps: 

1 . Consent to Participate: Participating instructors authorized the use of their 
submitted course materials in the creation of a composite Reference Course 
Profile. In addition, instructors granted the THECB permission to publish, in part 
or in whole, any of the documents that were subsequently incorporated into a 
Reference Course Profile. (See Appendix A to view a copy of the consent form.) 

2. Course Profile: Participating instructors provided general course information 
including course objectives, class size, grading policy, texts used, prerequisite 
and pathways courses, and percentage of students who enter their course well 
prepared. 

3. Course Ratings: Instructors were asked to rate the first three levels of the cross- 
disciplinary standards. The three top levels were included so as to be able to 
analyze the level of necessity of the cross-disciplinary knowledge and skills. For 
example, was an entire organizing component (the second level) not necessary 
for a CTE course, or just individual performance expectations within the 
organizing component? The fourth level of the CCRS, the performance 
indicators, are not standards per se, but examples of how the standards could be 
demonstrated and measured. As the intent of the performance indicators was for 
example purposes only, they were not included in the ratings analysis. 
Participating instructors completed an online rating form that asked them to 
answer the following question for each cross-disciplinary standard: “How 
necessary is this element in preparing students to succeed in my course?” 
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Respondents chose one of five options: most necessary, more necessary, less 
necessary, least necessary, or not necessary. After selecting a response option 
for each standard, instructors then selected one or more rationale statements to 
explain the reason they rated the item the way that they did. (See Appendix B for 
a list of scale items and rationale statements.) The rationale statements were 
included to provide greater clarity of understanding of responses. For example, 
an instructor might designate a standard as not necessary or least necessary tor 
one of several reasons. For example, the standard might not be necessary to 
succeed in the course because it was irrelevant to the subject area, or it might be 
covered in a subsequent course. The rationale statements were particularly 
valuable in interpreting the reasons why specific standards were found to be not 
well aligned. 

4. Additional Questions: Participating instructors also responded to a set of specific 
questions to collect data on common components of and current practices in 
entry-level Career and Technical Education courses. (See Appendix C for the list 
of additional questions.) 

5. Upload Course Materials: Participating instructors uploaded key course 
documents, including syllabi, assignments, assessments, grading rubrics and 
any other relevant materials. All identifying information was removed. 

Overall, instructors at 43 separate public two-year postsecondary institutions throughout 
the state of Texas completed course submissions. Table 5 presents an overview of the 
disposition of all nominated courses. 



Table 5: Final Course Status for All Nominated Courses 



Course Title 


Completed 

Course 

Nomination 


Partial 

Completed 

Course 

Submission 


Declined 

Participation 


No 

Response 


Total 


ACNT 1303 Introduction to 
Accounting 1 


25 


0 


1 


9 


35 


BMGT 1303 Principles of 
Management 


20 


0 


0 


11 


31 


DFTG 1309 Basic CAD 


20 


1 


0 


3 


24 


DFTG 1405 Technical Drafting 


15 


0 


0 


5 


20 


ITSC 1301 Introduction to 
Computers 


11 


1 


0 


8 


20 


ITSC 1401 Introduction to 
Computers 


9 


1 


0 


3 


13 


MRKG 1311 Principles of 
Marketing 


19 


0 


2 


6 


27 
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Course Title 


Completed 

Course 

Nomination 


Partial 

Completed 

Course 

Submission 


Declined 

Participation 


No 

Response 


Total 


POFI 1301 Computer Applications 1 


17 


0 


0 


1 


18 


POFT 1301 Business English 


21 


0 


0 


2 


23 


Total 


157 


3 


3 


48 


211 



Courses submitted from an individual institution ranged from 1 to 14. The average 
number of courses received from participating campuses was 3.7. Table 3 summarizes 
the distribution of course submissions by institution type and region. 



Table 6: Distribution of all Course Submissions by Region and Institution Type 



Region 


Community College 


Technical College 


Total 


Central 


27 


0 


27 


Gulf Coast 


20 


0 


20 


High Plains 


5 


0 


5 


Metroplex 


35 


0 


35 


Northwest 


5 


0 


5 


South 


12 


4 


16 


Southeast 


7 


8 


15 


Upper East 


13 


2 


15 


Upper Rio Grande 


4 


0 


4 


West 


15 


0 


15 


Total 


143 


14 


157 



Ratings 

To determine the level of alignment, the modal (most frequent) instructor response was 
determined for each individual cross-disciplinary standard. This approach is consistent 
with the methodology employed in the TCCRI Phase II study. Each standard could have 
a score of 1 through 5 (from most necessary to not necessary). Depending upon the 
level of necessity selected, instructors then selected rationale statements that best 
explained their responses. (See Appendix D for a list of scale items and rationale 
statements.) Items left blank by instructors were treated as missing data. The overall 
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results of the faculty ratings are presented in the next section. For a detailed course-by- 
course breakdown of the results, please refer to the original Phase I CTE Study. 

Results 

Overall, the findings from this study indicate that every OCRS cross-disciplinary 
standard is aligned with at least one of the nine CTE courses analyzed. The level of 
alignment (including standards deemed either necessary for preparation or covered in 
the course) between the full set of cross-disciplinary standards, and the nine CTE 
course titles analyzed ranges from 100 percent in DFTG 1405 Technical Drafting to 66 
percent in POFT 1301 Business English. While the level of alignment of the cross- 
disciplinary CCRS and any single course included varies, an examination across all 
CTE courses studied reveals high alignment between the cross-disciplinary skills across 
a range of typical entry-level CTE coursework. The overall alignment results are 
presented in Table 7. 



Table 7 ; Summary of Alignment of Cross-disciplinary CCRS Deemed Necessary or 
Taught in the CTE Courses 2 



Course Title 


Percent of Cross-Disciplinary Standards 
Necessary or Taught 


ACNT 1303 Introduction to Accounting 1 


81% 


BMGT 1303 Principles of Management 


84% 


DFTG 1309 Basic CAD 


84% 


DFTG 1405 Technical Drafting 


100% 


ITSC 1301 Introduction to Computers 


97% 


ITSC 1401 Introduction to Computers 


84% 


MRKG 1311 Principles of Marketing 


98% 


POFI 1301 Computer Applications 1 


76% 


POFT 1301 Business English 


66% 



2 May not total to 100 percent due to rounding error. 
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To determine the overall level of alignment, the modal ratings and rationale responses 
were combined. When looking at the specific rationales behind the instructor ratings, a 
pattern emerges. The typical (modal) rationale for all of the inconsistently aligned 
standards were described as being less necessary for successful preparation because 
the element was expected to be taught in the course. All of these inconsistently aligned 
standards were considered appropriate content. Since all of the inconsistently aligned 
standards were considered appropriate content for the courses, all aligned and 
inconsistently aligned standards were included in the overall results. 

Highly Aligned Organizing Components and Performance Expectations 

Of the 1 1 organizing components of the cross-disciplinary OCRS, five are highly aligned 
to CTE courses. Highly aligned is determined by alignment in each of the nine CTE 
courses included in this study. Table 8 below lists the five cross-disciplinary organizing 
components highly aligned across all CTE courses. 



Table 8: Organizing Components Highly Aligned Across All CTE Courses by Rank 



Rank 


Cross-Disciplinary 

Standard 


Total 

Responses 


Total 

"Most" 

Responses 


Total "More" 
Responses 


Total 
Aligned 
Responses 
("Most" or 
"More") 


Percent 

Aligned 

Responses 


1 . 


I.E. Work habits 


138 


72 


59 


131 


95% 


2. 


I.D. Academic 
behaviors 


137 


56 


67 


123 


90% 


3. 


I.F. Academic integrity 


139 


69 


49 


118 


85% 


4. 


II. E. Technology 


152 


63 


54 


117 


77% 


5. 


II. A. Reading across 
the curriculum 


150 


37 


75 


112 


75% 



Of the 45 performance expectations of the cross-disciplinary CCRS, 12 are highly 
aligned across the nine CTE courses. Table 9 below lists, in order of the strength of 
alignment, the highly aligned performance expectations. These aligned standards are 
split equally between Key Cognitive Skills and Foundational Skills, suggesting that each 
of these areas is equally valuable for preparation for success in CTE courses. 
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Table 9: Performance Expectations Highly Aligned Across All CTE Courses by Rank 



Rank 


Cross-Disciplinary Standard 


Total 

Responses 


Total 

"Most" 

Responses 


Total 

"More" 

Responses 


Total 
Aligned 
Responses 
("Most" or 
"More") 


Percent 

Aligned 

Responses 


1. 


I.E.1 . Work independently. 


145 


75 


59 


134 


92% 


2. 


I.D.2. Use study habits 
necessary to manage 
academic pursuits and 
requirements. 


144 


63 


66 


129 


90% 


3. 


I.D.4. Persevere to complete 
and master tasks. 


144 


80 


49 


129 


90% 


4. 


II.A.4. Identify the key 
information and supporting 
details. 


153 


54 


82 


136 


89% 


5. 


I.F.4. Understand and adhere 
to ethical codes of conduct. 


145 


83 


44 


127 


88% 


6. 


I.D.1. Self-monitor learning 
needs and seek assistance 
when needed. 


143 


57 


68 


125 


87% 


7. 


II. E. 4. Use technology 
appropriately. 


153 


66 


56 


122 


80% 


8. 


II.A.2. Use a variety of 
strategies to understand the 
meanings of new words. 


153 


39 


81 


120 


78% 


9. 


II.A.1 . Use effective 
prereading strategies. 


153 


32 


79 


111 


73% 


10. 


II. E. 3. Use technology to 
communicate and display 
findings in a clear and 
coherent manner. 


153 


56 


54 


110 


72% 


11. 


II.E.1 . Use technology to 
gather information. 


153 


47 


57 


104 


68% 


12. 


I.F.1 . Attribute ideas and 
information to source materials 
and people. 


144 


45 


48 


93 


65% 
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Significance 

The findings of this study do indicate that it is reasonable to assume the cross- 
disciplinary skills are necessary for career readiness. It is predictable that some of the 
individual courses examined did have lower levels of alignment (such as POFT 1301 
Business English). Individual courses are often narrow in scope and not representative 
of the full constellation of knowledge and skills developed in a full technical program 
associated with a given career path. This study tested the assumption that Key 
Cognitive and Foundational Skills are necessary for CTE preparation, and the data 
strongly suggest that students do need to possess a full array of cognitive and 
behavioral skills and strategies to be successful in a range of CTE courses. 

The snapshot of CTE courses examined provides an informative picture of the 
knowledge and skills currently necessary for preparation in a range of entry-level CTE 
courses. For example, five of the CCRS cross-disciplinary organizing components were 
found to be highly aligned across all the CTE courses included in this study. These five 
organizing components include (in rank of order of alignment level): work habits, 
academic behaviors, academic integrity, technology, and reading across the curriculum. 

This sort of clear finding can help inform high school faculty immediately regarding the 
kinds of skills that are important to emphasize in courses containing students who seem 
likely to go on to CTE programs upon completion of high school. The identified 
organizing components could be emphasized throughout the four years of high school in 
all classes, with special attention to their implementation in high school CTE courses. 

For postsecondary instructors and administrators, this study offers insight into current 
expectations and practice in a range of CTE courses. The results of this study are 
augmented by the CTE Reference Course Profiles that were also generated based upon 
the results of this study and a range of additional course documents collected. The 
results of this study, along with the Profiles, can potentially serve as a reference point 
for postsecondary instructors and course planners who have responsibility for entry- 
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level CTE courses. While instructors retain academic freedom for the way they teach 
their courses, the CTE Reference Course Profiles can provide an additional point of 
comparison instructors can use when creating or refining entry-level courses. The goal 
is to assist all CTE postsecondary programs to ensure that the entry-level courses they 
offer are aligned with the CCRS, contain appropriate content, and are cognitively 
challenging. 

Finally, the study also suggests a potential gap between the current level of preparation 
of entering CTE students and an optimal level of preparation for future students. If the 
goal is to have more students attend and succeed in CTE programs, then proficiency 
with the Key Cognitive and Foundational Skills addressed by the cross-disciplinary 
standards could fundamentally enhance their success and enable CTE instructors to 
teach more efficiently and effectively. The end result would be courses in which 
students learn more and in which more students succeed. 
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Project #5: 

Alignment Between the CCRS and Placement Tests 
Commonly Used in Texas Postsecondary Institutions 

Overview 

Although the number of high school graduates pursuing higher education has increased 
during the last 30 years, between 30 percent and 60 percent of college freshmen 
require remedial education (NCES, 2004). A student may be college-eligible— that is, 
able to meet college admissions requirements— without being college ready (Conley, 

2005). To be college ready, a student must attain a level of preparation to enroll and 
succeed— without remediation— in credit-bearing general education courses at the 
postsecondary level (Conley, 2007). Not only does remedial education cost an 
estimated $1 billion annually, but students requiring it are far less likely to graduate 
(ACT, 2005). As a result, significant attention is now directed to understanding and 
remedying the gap between what high school graduates know and are capable of and 
what entry-level college students need to know and are capable of. 

Addressing the lack of preparedness of high school graduates for college-level 
coursework requires identifying the gaps in content between what is taught in high 
school and what is expected of students in college. It also requires appropriate and 
comprehensive measures for educators to monitor progress toward closing this gap. 

One option for monitoring the developing college preparedness of high school students 
is through state assessments. Required state high school assessments often shape 
what gets emphasized in classrooms. Many states do not yet explicitly align the content 
knowledge and skills assessed by the state tests with those required for college 
success. Conley (2003) and Brown and Conley (2007) evaluated state tests and found 
that the content of state high school tests is inconsistently and only moderately aligned 
with college readiness standards in 20 states in mathematics and ELA. They find 
slightly more alignment with the ELA exams than with the mathematics exams, yet the 
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ELA exams align poorly or not at all in areas requiring higher levels of thinking, and the 
mathematics exams have the highest alignment in basic skills content areas. The 
findings describe the gap between student learning expectations and what they are 
expected to know to be ready for college and provide evidence that state tests are not 
aligned well enough with college readiness standards to be useful for providing 
feedback to high school students and teachers concerning college readiness. 

Consequently, many students graduating from high school and passing their state’s 
high-stakes assessments are not sure how well prepared they are for postsecondary 
coursework. 

College admissions and placement tests, then, become more important means for 
assessing college preparedness. Colleges and universities use scores on placement 
exams for a variety of purposes but most often for determining whether a student should 
have to take a remedial course before being allowed access to credit-bearing, college- 
level courses. Even though these exams are explicitly college-focused, evidence 
suggests that they are only moderately aligned with college readiness standards. Brown 
and Niemi (2007), for example, investigated the alignment of high school test content 
and community college test placement content (ACCUPLACER and COMPASS) and 
find that the content of ELA tests are adequately aligned but that mathematics tests are 
aligned only in depth-of-knowledge consistency. This study focuses on community 
college expectations, however, and the results may not generalize fully to four-year 
institutions. Achieve, Inc. (2007), also analyzes the alignment of admissions and 
placement exams in mathematics and English (ACT, SAT, ACCUPLACER, COMPASS, 

THEA) with the American Diploma Project (ADP) benchmarks. Their analysis, which 
may be considered more generalizable, finds that the ACT and SAT are generally more 
demanding than placement tests and demonstrate better coverage of a range of content 
knowledge. All the tests fell short of measuring the full set of knowledge and skills 
encompassed by the ADP benchmarks. Overall, the study finds that the tests place too 
much emphasis on low-level content taught earlier in a student’s high school education 
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or even in middle school. 

Texas is a leader in the movement to align high school curricula and assessments with 
college readiness expectations and was one of the first states to adopt a specific and 
comprehensive definition of college readiness describing essential knowledge and skill 
standards for all students in the state. 

With the foundation of college and career readiness standards firmly in place, Texas is 
prepared to take the next steps in improving college readiness: to ascertain the 
alignment of postsecondary placement with the CCRS. By doing so, the THECB can 
ascertain the alignment of its measures of placement with the state standards and can 
modify placement procedures and practices so that incoming students who focus on 
meeting the CCRS while in high school are placed into college courses accordingly. 

The purpose of the Texas Test Alignment Project (TAP) is to examine the alignment of 
THECB-approved college admission and placement tests used by Texas institutions of 
higher education to measure college readiness with the CCRS. As such, the goal of the 
current study is not to identify the best test to assess the CCRS, but to ascertain the 
discrepancy between current practice and the desired goal of complete measurement of 
the CCRS. This study addresses the following research objectives: 

• To what extent do college admissions and placement tests assess the 
knowledge and skills specified in college and career readiness standards? 

• Are any areas of college and career readiness standards not assessed by 
placement and admissions tests? 

• How do the tests compare to college and career ready standards in terms of rigor 
and cognitive demand? 

Methodology 

Test items from the six THECB-approved placement and admissions tests used by 
Texas institutions of higher education to measure college readiness were obtained for 
the study. Content experts in mathematics and ELA first rated the rigor and depth of 
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knowledge required by the CCRS. The CCRS are organized into three levels: key 
content areas, organizing components, and performance expectations. The reviewers 
made their judgments at the level of performance expectations by rating the cognitive 
complexity of each performance expectation and then determining whether each test 
item from each of the designated tests corresponds with one or more performance 
expectations. Every reviewer determined the relationship between each test item and 
the corresponding performance expectations. The results show the average cognitive 
complexity ratings and describe the relationship between performance expectations and 
test item’s content (their categorical concurrence) and, when they match, whether rigor 
and depth of knowledge between the performance expectation and the test item is 
appropriate. 

Researchers have developed many ways to assess alignment (Rothman, 2004; Webb, 

1997, 1999; Porter, 2002; ACHIEVE, 2000, 2001, and 2003; Wixson, Fisk, Dutro, and 
McDaniel, 2002). Webb’s method is one of the more widely used ways to determine 
alignment between test items and educational standards. His approach however, is 
often modified by alignment researchers (Impara, 2001). 

This study also is based on Webb’s methods, but were modified to better address the 
project’s objectives. The primary deviation is to compute each metric across raters 
rather than by computing alignment metrics for each rater and then averaging them. 

Although the estimates may be slightly less sensitive to differences in reviewer ratings, 
they highlight the alignment between each of the tests and the CCRS— as this 
alignment is a core objective of the study. 

• Categorical concurrence determines the extent that test items correspond to 
the CCRS. Tests with more than six items assessing a standard are considered 
to demonstrate categorical concurrence with that standard. 

• Depth-of-knowledge (DOK) consistency is measured by comparing the DOK 
level required of the test item (as determined by the reviewers) with the DOK 
required to meet the standard (as determined by reviewers). The recommended 
benchmark is that at least one-half the assessment items are at or above the 
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DOK required by the corresponding standard. 

• Range of knowledge is the extent to which the CCRS are assessed by items on 
a test. This criterion is typically measured as the average percent of standards 
addressed by one or more items on each test (across reviewers). 

Webb includes a fourth measure, balance of representation, to describe the extent that 
assessment items are evenly distributed across learning objectives. Previous research 
provides evidence that this is highly redundant with the range of knowledge measure 
(e.g., Brown and Conley, 2007). Furthermore, when there is great variation in test 
distribution (e.g., the number of items provided to represent each test), as in the current 
study, this distribution metric is more likely to be skewed and not be directly comparable 
across tests. For these reasons, balance of representation is not reported here. 

Results 

Reviewer Agreement 

As is typical with alignment studies, the ratings of six experts were used as the basis for 
alignment computations and decisions. Six is the standard number of reviewers required 
in similar alignment studies to obtain sufficient reliability (Herman, Webb and Zuniga, 

2005; Webb 1997, 1999, and 2002). Consistent with the treatment of rigor ratings and 
cognitive demand ratings as quasi-quantitative measures (rather than as strictly 
categorical measures), a generalizability analysis was conducted with items and 
standards crossed by reviewers (Shavelson and Webb, 1991). 

The absolute error variance indicates the consistency of item-and-standard ratings 
across reviewers. The phi coefficient, called the index of dependability { Shavelson and 
Webb, 1991), can be considered a reliability-like coefficient for absolute decisions 
(Herman, Webb and Zuniga, 2007; Mushquash and O’Connor, 2006; Thompson, 2003). 

The phi coefficient in this context represents decision consistency over parallel rating 
situations. For example, a phi coefficient of 0.80 indicates that, if the same study was 
recreated and repeated over multiple occasions, one could expect the same pattern of 
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ratings 80% of the time. Coefficients are interpreted similarly to reliability coefficients in 
that values of 0.80 and higher indicate high generalizability. 

Table 10 shows the generalizability results for the rigor and cognitive-demand ratings of 
the mathematics and English items across assessments. The phi coefficients for the 
cognitive-demand ratings are close to or above the conventional 0.80 criterion for 
reliability. The phi coefficients for the rigor ratings were lower than those for the 
cognitive-demand ratings, with none of the coefficients reaching the conventional 
criterion for reliability. The six reviewers obtained an acceptable level of dependability 
for estimating mathematics and ELA items’ level of cognitive demand, but were much 
less consistent in their rating of the level of rigor. 

Table 10. G-Study Coefficients for Mathematics and ELA Items across Test 



Subject 


Items 


Cognitive Demand 


Rigor 


Phi Coefficients 


Absolute Error Variance 


Phi Coefficients 


Absolute Error Variance 


Mathematics 


1,460 


0.859 


0.035 


0.505 


0.007 


ELA 


1,239 


0.703 


0.053 


0.446 


0.02 



Table 1 1 shows the generalizability results for the cognitive demand ratings and rigor 
ratings of the mathematics and ELA CCRS performance expectations. As with the item 
ratings, the phi coefficients for the performance expectation cognitive-demand ratings 
are close to or above the conventional 0.80 criterion for reliability. The phi coefficients 
for the rigor ratings were lower than those for the cognitive demand ratings, with none of 
the coefficients reaching the conventional criterion for fully satisfactory reliability. These 
results indicate that the six reviewers did reach an acceptable level of dependability for 
estimating mathematics and ELA standards’ level of cognitive demand but not their level 
of rigor. 
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Table 1 1. G-Study Coefficients for Mathematics and ELA OCRS Performance 
Expectations Ratings 



Subject 


Standards 


Cognitive Demand 


Rigor 


Phi 

Coefficients 


Absolute Error 
Variance 


Phi 

Coefficients 


Absolute Error 
Variance 


Mathematics 


115 


0.855 


0.100 


0.566 


0.038 


ELA 


119 


0.724 


0.095 


0.556 


0.060 



Overall, these results indicate higher rater reliability for mathematics item ratings and 
standard ratings than for ELA item ratings and standard ratings. Additionally, the results 
indicate higher rater reliability for cognitive-demand ratings than for rigor ratings. Results 
are consistent with findings from other similar studies using the same number of 
reviewers (Herman, Webb, and Zuniga, 2005, Brown and Conley, 2007, Brown and 
Niemi, 2007). Interestingly, Herman, Webb, and Zuniga also reported that ratings of 
cognitive demand were more reliable than were ratings of centrality (similar to rigor in 
that centrality evaluated the extent that a standard was essential to a topic). 

A D-study (decision study) was also conducted, which determined the gains in reliability 
expected if the number of reviewers was increased in future studies. The results 
indicate that, in most cases, increasing the number of reviewers from 6 to 10-15 would 
result in only moderate gains in reliability (as measured by the phi coefficient). 

Findings 

Study findings are described below beginning with a summary of the rigor and cognitive 
demand of the CCRS and the tests, including: 

• Average rigor and cognitive demand of CCRS organizing components 

• Average rigor and cognitive demand of tests, across items 

Also described is the alignment between the test items and the CCRS, specifically: 

• The total number of matches or hits for each performance expectation 

• Zero-match performance expectations, or those not assessed by any of the tests 
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• Categorical concurrence, modified to describe when the number of items 
assessing a performance expectation on each test is six or more for mathematics 
and ELA tests 

• Depth-of-knowledge consistency for mathematics and ELA tests, describing 
average differences in rigor and cognitive-demand ratings 

• Range of knowledge for mathematics and ELA tests identifying the total number 
of matches for each performance expectation by test 

Rigor and Cognitive Demand 

Texas Career and College Readiness Standards Rigor and Cognitive Demand 

Reviewers provided rigor and cognitive-demand ratings for each of the CCRS 
performance expectations. Each reviewer’s ratings were averaged across the expanded 
performance expectations and then averaged across reviewers to generate rigor and 
cognitive-demand scores for each item and then for each performance expectation. 

Table 12 provides the average of reviewers’ rigor and cognitive-demand ratings across 
performance expectation for each organizing component of the CCRS cross-disciplinary 
skills. On average, the reviewers found the cross-disciplinary skills to be at or below the 
level at which an entry-level college student should perform (between 1 and 2 on the 
rigor scale). Reviewers rate the key cognitive strategies at a higher level of rigor and 
cognitive demand than the foundational skills. Note the similar patterns in findings 
between rigor and cognitive demand; however, given the higher reliability of the 
cognitive demand ratings, somewhat greater consideration should be given to cognitive 
demand ratings than to rigor ratings. 



Table 12. Average Rigor and Cognitive Demand for Cross-Disciplinary Skills by 
Organizing Component 



Organizing Component 


Average Rigor 

(3-point scale) 


Average Cognitive Demand 

(4-point scale) 


Mean 


N 


SD 


Mean 


N 


SD 


Key Cognitive Strategies 














Intellectual curiosity 


2.04 


2 


0.177 


3.29 


2 


0.059 


Reasoning 


1.92 


4 


0.180 


3.35 


4 


0.299 
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Organizing Component 


Average Rigor 

(3-point scale) 


Average Cognitive Demand 

(4-point scale) 


Mean 


N 


SD 


Mean 


N 


SD 


Problem solving 


1.89 


3 


0.192 


3.33 


3 


0.220 


Academic behaviors 


1.40 


4 


0.125 


2.38 


4 


0.285 


Work habits 


1.13 


2 


0.059 


2.46 


2 


0.059 


Academic integrity 


1.60 


4 


0.197 


2.58 


4 


0.642 


Overall Key Cognitive Strategies Mean 


1.67 


19 


0.329 


2.88 


19 


0.549 


Foundational Skills 














Reading across the curriculum 


1.23 


8 


0.165 


2.05 


8 


0.478 


Writing across the curriculum 


1.14 


3 


0.127 


2.00 


3 


0.500 


Research across the curriculum 


1.59 


8 


0.225 


2.59 


8 


0.384 


Use of data 


1.81 


3 


0.268 


2.94 


3 


0.459 


Technology 


1.29 


4 


0.144 


1.96 


4 


0.567 


Overall Foundational Skills Mean 


1.41 


26 


0.287 


2.30 


26 


0.554 



Note: Averages were computed for each performance expectation across reviewers and then were averaged across 
performance expectations within each organizing component. “N” refers to the number of performance expectations 
within each organizing component. The standard deviations (SD) describe the variability of the performance 
expectations within organizing components. Shaded cells indicate the organizing components with the highest 
average rigor or cognitive demand ratings. 

Within the cross-disciplinary skills, the key cognitive strategies were on average, more 
rigorous and cognitively demanding than the foundational skills. 

Table 13 provides the average of reviewers’ rigor and cognitive-demand ratings across 
performance expectations for each of the mathematics organizing components. The 
rigor ratings are slightly lower for mathematics than for the cross-disciplinary skills. On 
average, the reviewers rated the mathematics standards as below the level at which an 
entry-level college student should be expected to perform (near 1 on the rigor scale). 



Table 13. Average Rigor and Cognitive Demand for Mathematics by Organizing 
Component 



Organizing Component 


Average Rigor 

(3-point scale) 


Average Cognitive Demand 

(4-point scale) 


Mean 


N 


SD 


Mean 


N 


SD 


Numeric Reasoning 














Number representation 


1.08 


2 


0.118 


1.00 


2 


0.000 


Number operations 


1.17 


1 




1.00 


1 




Number sense and number concepts 


1.00 


1 




1.83 


1 
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Organizing Component 


Average Rigor 

(3-point scale) 


Average Cognitive Demand 

(4-point scale) 


Mean 


N 


SD 


Mean 


N 


SD 


Overall Numeric Reasoning 


1.08 


4 


0.096 


1.21 


4 


0.417 


Algebraic Reasoning 














Expressions and equations 


1.17 


1 




1.67 


1 




Manipulating expression 


1.00 


1 




1.17 


1 




Solving equations, inequalities, and systems of 
equations 


1.08 


2 


0.118 


1.58 


2 


0.589 


Representations 


1.08 


2 


0.118 


2.00 


2 


0.000 


Overall Algebraic Reasoning 


1.08 


6 


0.091 


1.67 


6 


0.408 


Geometric Reasoning 














Figures and their properties 


1.11 


3 


0.192 


1.78 


3 


1.072 


Transformations and symmetry 


1.11 


3 


0.192 


1.61 


3 


0.770 


Connections between geometry and other 
mathematical content strands 


1.17 


3 


0.167 


1.83 


3 


0.167 


Logic and reasoning in geometry 


1.17 


2 


0.000 


2.42 


2 


0.825 


Overall Geometric Reasoning 


1.14 


11 


0.146 


1.86 


11 


0.710 


Measurement Reasoning 














Measurement involving physical and natural 
attributes 


1.00 


1 




1.17 


1 




Systems of measurement 


1.00 


2 


0.000 


1.00 


2 


0.000 


Measurement involving geometry and algebra 


1.06 


3 


0.096 


1.22 


3 


0.385 


Measurement involving statistics and probability 


1.25 


2 


0.118 


1.92 


2 


1.296 


Overall Measurement Reasoning 


1.08 


8 


0.126 


1.33 


8 


0.649 


Probabilistic Reasoning 














Counting principles 


1.17 


1 




1.17 


1 




Computation and interpretation of probabilities 


1.33 


2 


0.236 


1.50 


2 


0.471 


Overall Probabilistic Reasoning 


1.28 


3 


0.192 


1.39 


3 


0.385 


Statistical Reasoning 














Data collection 


1.67 


1 




2.50 


1 




Describe data 


1.04 


4 


0.083 


1.54 


4 


0.479 


Read, analyze, interpret, and draw conclusions 
from data 


1.50 


4 


0.136 


2.67 


4 


0.304 


Overall Statistical Reasoning 


1.31 


9 


0.282 


2.15 


9 


0.674 


Functions 














Recognition and representation of functions 


1.00 


2 


0.000 


1.17 


2 


0.236 


Analysis of functions 


1.08 


2 


0.118 


2.25 


2 


0.589 


Model real world situations with functions 


1.33 


2 


0.471 


2.33 


2 


0.707 


Overall Functions 


1.14 


6 


0.267 


1.92 


6 


0.721 
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Organizing Component 


Average Rigor 

(3-point scale) 


Average Cognitive Demand 

(4-point scale) 


Mean 


N 


SD 


Mean 


N 


SD 


Problem Solving and Reasoning 














Mathematical problem solving 


1.30 


5 


0.139 


3.10 


5 


0.418 


Logical reasoning 


1.50 


2 


0.236 


3.50 


2 


0.471 


Real world problem solving 


1.33 


3 


0.167 


2.83 


3 


0.601 


Overall Problem Solving and Reasoning 


1.35 


10 


0.166 


3.10 


10 


0.492 


Connections 














Connections among the strands of mathematics 


1.25 


2 


0.118 


2.67 


2 


0.000 


Connections of mathematics to nature, real-world 
situations, and everyday life 


1.50 


3 


0.167 


2.50 


3 


0.167 


Overall Connections 


1.40 


5 


0.190 


2.57 


5 


0.149 


Communication and Representation 














Language, terms, and symbols of mathematics 


1.06 


3 


0.096 


2.22 


3 


0.674 


Interpretation of mathematical work 


1.25 


2 


0.118 


2.25 


2 


0.354 


Presentation and representation of mathematical 
work 


1.33 


3 


0.167 


2.61 


3 


0.509 


Overall Communication and Representation 


1.21 


8 


0.173 


2.38 


8 


0.510 



Note: Averages were computed for each performance expectation across reviewers and then were averaged across 
performance expectations within each organizing component. “N” refers to the number of performance expectations 
within each organizing component. The standard deviations (SD) describe the variability of the performance 
expectations within organizing components. Shaded cells indicate the organizing components with the highest 
average rigor or cognitive demand ratings. 

The organizing components with the highest average cognitive demand ratings belong 
to the statistical reasoning, problem solving and reasoning, and statistical reasoning key 
content areas, including the connections and problem solving and reasoning, data 
collection, read, analyze, interpret, and draw conclusions from data, logical reasoning, 
and connections of mathematics to nature, real-world situations, and everyday life 
organizing components. The organizing components with the highest cognitive demand 
ratings tend also to be high in rigor. 

Table 14 provides the average of reviewers’ rigor and cognitive-demand ratings across 
performance expectations for each of the ELA organizing components. On average, the 
ELA standards are at the level an entry-level college student should perform (close to 2 
on the rigor scale). 
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Table 14. Average Rigor and Cognitive Demand for English/Language Arts by Organizing 
Component 



Organizing Component 


Average 

Rigor 


Average 

Cognitive Demand 


Mean 


N 


SD 


Mean 


N 


SD 


Writing 














Compose a variety of texts that demonstrate clear focus, the 
logical development of ideas in well-organized paragraphs, and 
the use of appropriate language that advances the author’s 
purpose 


1.83 


5 


0.199 


2.42 


5 


0.391 


Overall Writing 


1.83 


5 


0.199 


2.42 


5 


0.391 


Reading 














Locate explicit textual information and draw complex 
inferences, analyze, and evaluate the information within and 
across texts of varying lengths 


1.59 


11 


0.277 


2.35 


11 


0.490 


Understand new vocabulary and concepts and use them 
accurately in reading, speaking, and writing 


1.22 


3 


0.096 


1.50 


3 


0.333 


Describe, analyze, and evaluate information within and across 
literary and other texts from a variety of cultures and historical 
periods 


1.83 


4 


0.381 


2.51 


4 


0.790 


Explain how literary and other texts evoke personal experience 
and reveal character in particular historical circumstances 


1.92 


2 


0.589 


2.75 


2 


0.825 


Overall Reading 


1.62 


20 


0.355 


2.30 


20 


0.637 


Research 














Formulate topic and questions 


1.53 


3 


0.127 


2.47 


3 


0.173 


Select information from a variety of sources 


1.81 


4 


0.336 


2.60 


4 


0.448 


Produce and design a document 


1.75 


2 


0.118 


2.71 


2 


0.059 


Overall Research 


1.70 


9 


0.257 


2.58 


9 


0.303 



Note: Averages were computed for each performance expectation across reviewers and then were averaged across 
performance expectations within each organizing component. Standard deviations describe the variability of the 
performance expectations within organizing components. Shaded cells indicate the organizing components with the 
highest average rigor or cognitive demand ratings. 

For ELA, writing and research are rated as more rigorous and cognitively demanding 
than reading, and the average rigor and cognitive demand for the ELA organizing 
components are higher than for the mathematics and the cross-disciplinary skill 
organizing components. 

Test Rigor and Cognitive Demand 

The reviewers found tests overall to be nearly identical in their level of rigor, which on 
average was below the level of the OCRS and below the level at which an entry-level 
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college student should perform. This should not come as a surprise because 
admissions and placement tests must cover a range of knowledge and skill levels to 
uncover test takers who are below the level of a credit-bearing college course. To do so 
requires at least some items that are below college level. This explains why, in most 
cases, the average test items were less rigorous than the standards. 

Table 15 describes the overall average rigor ratings for each of the tests by subject. 



Table 15. Average Rigor for Mathematics and English/Language Arts by Test 



Test 


Average Mathematics Item Rigor 


Average ELA Item Rigor 


(Averaged across rater and items) 


(Averaged across rater and items) 


Mean 


N 


SD 


Mean 


N 


SD 


A 


1.08 


248 


0.139 


1.21 


162 


0.142 


B 


1.03 


180 


0.098 


1.16 


347 


0.179 


C 


1.02 


528 


0.070 


1.08 


240 


0.110 


D 


1.06 


351 


0.146 


1.08 


240 


0.158 


E 


1.08 


100 


0.124 


1.21 


167 


0.178 


F 


1.05 


53 


0.107 


1.26 


83 


0.190 



There is greater variability in the overall cognitive-demand ratings. Table 16 describes 
the average cognitive-demand ratings for each test by subject. 



Table 16. Average Cognitive Demand for Mathematics and English/Language Arts by Test 



Test 


Average Mathematics Item Cognitive Demand 


Average ELA Item Cognitive Demand 


(Averaged across rater and item) 


(Averaged across rater and item) 




Mean 


N 


SD 


Mean 


N 


SD 


A 


1.39 


248 


0.376 


1.64 


162 


0.352 


B 


1.84 


180 


0.487 


1.37 


347 


0.427 


C 


1.50 


528 


0.502 


1.28 


240 


0.341 


D 


1.56 


351 


0.475 


1.42 


240 


0.389 


E 


2.05 


100 


0.369 


1.46 


167 


0.42 


F 


1.73 


53 


0.463 


1.59 


83 


0.421 



Within the ELA tests, the constructed-response writing prompts received higher rigor 
and cognitive demand ratings than did the multiple-choice reading items (with some 
variability in ratings by test). These were also more cognitively demanding and were the 
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only items to consistently require the highest level of cognitive demand. 

Alignment Between Test Items and CCRS 

Test Coverage of CCRS 

Reviewers identified a total of 57,391 hits or matches between the 2,699 test items and 
the 149 CCRS performance expectations. As expected, the test items assessed multiple 
standards, ranging from 15 to 34. Across tests, the average number of matches per test 
item ranged from 18 to 34 for mathematics and from 15 to 18 for ELA. 

Most- and Least-Assessed CCRS Organizing Components 

The most hits occurred for the key content areas of writing, problem solving and 
reasoning, algebraic reasoning, and foundational skills. The standards with the fewest 
test items assessing them are research, probabilistic reasoning, and statistical 
reasoning. 

Zero-Match Expectations 

Expectations not assessed by any test are called zero-match performance expectations. 

Table 17 identifies the CCRS items not assessed by a single test item on any of the 
tests. 



Table 17. Zero-Match CCRS Performance Expectations 



Key Content 


Organizing Components 


Performance Expectation 
Not Assessed by a Single Test Item 


Mathematics: 

Statistical 

Reasoning 


A. Data collection 


1. Plan a study 


C. Read, analyze, interpret, and draw 
conclusions from data 


3. Analyze relationships between paired data using 
spreadsheets, graphing calculators, or statistical 
software 


4. Recognize reliability of statistical results 


ELA: 

Reading 


A. Locate explicit textual information 
and draw complex inferences, analyze, 
and evaluate the information within and 
across texts of varying lengths 


8. Compare and analyze how generic features are 
used across texts 


B. Understand new vocabulary and 
concepts and use them accurately in 
reading, speaking, and writing 


2. Apply knowledge of roots and affixes to infer the 
meanings of new words 


3. Use reference guides to confirm the meanings of 
new words or concepts 
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Key Content 


Organizing Components 


Performance Expectation 
Not Assessed by a Single Test Item 




C. Describe, analyze, and evaluate 
information within and across literary 
and other texts from a variety of cultures 
and historical periods 


1 . Read a wide variety of texts from American, 
European, and world literatures 




3. Analyze works of literature for what they suggest 
about the historical period and cultural contexts in 
which they were written 




D. Explain how literary and other texts 
evoke personal experience and reveal 
character in particular historical 
circumstances 


2. Analyze the influence of myths, folktales, fables, 
and classical literature from a variety of world 
cultures on later literature and film 






1. Formulate research questions 




A. Formulate topic and questions 


2. Explore a research topic 


ELA: 


3. Refine research topic and devise a timeline for 
completing work 


Research 


B. Select information from a variety of 
sources 


4. Use source material ethically 




C. Produce and design a document 


1 . Design and present an effective product 




2. Use source material ethically 



Note: The OCRS include “Use source material ethically” in both the select information from a variety of sources and 
produce and design a document organizing components. 

Many of these appear to be skills best assessed by means other than multiple-choice or 
even constructed-response type test items: plan a study, use reference guides to 
confirm the meanings of new words, formulate research questions, explore a research 
topic, use source material ethically, etc. These are not included as in subsequent 
descriptive tables. However, they are counted when computing measures where the 
total number of performance expectations is a denominator, such as with categorical 
concurrence. 



Categorical Concurrence 

Categorical concurrence describes the degree to which the items in a test correspond 
with one or more of the CCRS. The criterion of categorical concurrence is met if the 
same categories of content appear in both the assessment and the standards; to 
produce an acceptable level of reliability for assessment scores, Webb recommends 
that at least six items assess each performance expectation. Table 18 describes the 
percent of performance expectations demonstrating categorical concurrence across 
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rater (i.e., that have at least six items assessing that performance expectation) within 
each organizing component for each mathematics test. 



Table 18. Categorical Concurrence by Key Content for Mathematics Tests 



Key Content 


Organizing Component 

(Number of 

performance expectations) 


Percent of Performance Expectations 
that Reached Categorical Concurrence 
for this Test in Mathematics 


A 


B 


C 


D 


E 


F 


Numeric Reasoning 


A. Number representation (2) 


50% 


50% 


100% 


50% 


50% 


0% 


B. Number operations (1) 


100% 


100% 


100% 


100% 


100% 


100% 


C. Number sense and number 
concepts (1) 


100% 


100% 


100% 


100% 


0% 


0% 


Numeric Reasoning Overall 


83% 


83% 


100% 


83% 


50% 


33% 


Algebraic Reasoning 


A. Expressions and equations (1) 


100% 


100% 


100% 


100% 


100% 


0% 


B. Manipulating expression (1) 


100% 


100% 


100% 


100% 


100% 


100% 


C. Solving equations, inequalities, 
and systems of equations (2) 


100% 


50% 


100% 


100% 


100% 


50% 


D. Representations (2) 


100% 


100% 


100% 


100% 


100% 


100% 


Algebraic Reasoning Overall 


100% 


88% 


100% 


100% 


100% 


63% 


Geometric Reasoning 


A. Figures and their properties (2) 


100% 


100% 


100% 


100% 


66% 


66% 


B. Transformations and symmetry (3) 


0% 


66% 


33% 


33% 


33% 


0% 


C. Connections between geometry 
and other mathematical content 
strands (3) 


66% 


66% 


66% 


66% 


66% 


66% 


D. Logic and reasoning in geometry 
(2) 


0% 


100% 


100% 


100% 


0% 


0% 


Geometric Reasoning Overall 


42% 


83% 


75% 


75% 


41% 


33% 


Measurement 

Reasoning 


A. Measurement involving physical 
and natural attributes (1) 


100% 


0% 


0% 


0% 


0% 


0% 


B. Systems of measurement (2) 


0% 


50% 


100% 


0% 


0% 


0% 


C. Measurement involving geometry 
and algebra (3) 


66% 


66% 


66% 


100% 


66% 


0% 


D. Measurement involving statistics 
and probability (2) 


0% 


0% 


0% 


0% 


0% 


0% 


Measurement Reasoning Overall 


42% 


29% 


42% 


25% 


17% 


0% 


Probabilistic 

Reasoning 


A. Counting principles (1) 


0% 


0% 


0% 


0% 


100% 


0% 


B. Computation and interpretation of 
probabilities (2) 


0% 


0% 


0% 


0% 


0% 


0% 


Probabilistic Reasoning Overall 


0% 


0% 


0% 


0% 


50% 


0% 


Statistical Reasoning 


A. Data collection (1) 


0% 


0% 


0% 


0% 


0% 


0% 


B. Describe data (4) 


0% 


25% 


25% 


25% 


100% 


0% 
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Key Content 


Organizing Component 

(Number of 

performance expectations) 


Percent of Performance Expectations 
that Reached Categorical Concurrence 
for this Test in Mathematics 


A 


B 


c 


D 


E 


F 


C. Read, analyze, interpret, and draw 
conclusions from data (4) 


0% 


0% 


0% 


0% 


25% 


0% 


Statistical Reasoning Overall 


0% 


13% 


13% 


13% 


63% 


0% 


Functions 


A. Recognition and representation of 
functions (2) 


50% 


0% 


50% 


50% 


0% 


0% 


B. Analysis of functions (2) 


100% 


100% 


100% 


100% 


100% 


50% 


C. Model real world situations with 
functions (2) 


50% 


100% 


50% 


100% 


100% 


100% 


Functions Overall 


67% 


67% 


67% 


83% 


67% 


50% 


Problem Solving and 
Reasoning 


A. Mathematical problem solving (5) 


80% 


80% 


80% 


80% 


80% 


80% 


B. Logical reasoning (2) 


0% 


50% 


50% 


50% 


50% 


50% 


C. Real world problem solving (3) 


33% 


100% 


33% 


33% 


33% 


33% 


Problem Solving and 
Reasoning Overall 


38% 


77% 


54% 


54% 


54% 


54% 


Communication and 
Representation 


A. Language, terms, and symbols of 
mathematics (3) 


66% 


100% 


100% 


100% 


66% 


66% 


B. Interpretation of mathematical 
work (2) 


50% 


100% 


50% 


100% 


50% 


100% 


C. Presentation and representation 
of mathematical work (3) 


0% 


0% 


33% 


33% 


33% 


33% 


Communication and 
Representation Overall 


39% 


67% 


61% 


78% 


50% 


66% 


Connections 


A. Connections among the strands of 
mathematics (2) 


50% 


100% 


50% 


50% 


50% 


50% 


B. Connections of mathematics to 
nature, real-world situations, and 
everyday life (3) 


33% 


66% 


33% 


33% 


33% 


33% 


Connections Overall 


42% 


83% 


42% 


42% 


42% 


42% 


Key Cognitive 
Strategies 


A. Intellectual curiosity (2) 


0% 


0% 


0% 


0% 


0% 


0% 


B. Reasoning (4) 


0% 


0% 


0% 


0% 


0% 


0% 


C. Problem solving (3) 


66% 


66% 


66% 


33% 


33% 


66% 


D. Academic behaviors (4) 


0% 


0% 


0% 


0% 


0% 


0% 


E. Work habits (2) 


0% 


0% 


0% 


0% 


0% 


0% 


F. Academic Integrity (4) 


0% 


0% 


0% 


0% 


0% 


0% 


Key Cognitive Strategies Overall 


11% 


11% 


11% 


6% 


6% 


11% 


Foundational Skills 


A. Reading across the curriculum (8) 


12% 


12% 


25% 


12% 


12% 


12% 


B. Writing across the curriculum (3) 


0% 


0% 


0% 


0% 


0% 


0% 
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Key Content 


Organizing Component 

(Number of 

performance expectations) 


Percent of Performance Expectations 
that Reached Categorical Concurrence 
for this Test in Mathematics 


A 


B 


C 


D 


E 


F 


C. Research across the curriculum 
(8) 


0% 


0% 


0% 


0% 


0% 


0% 


D. Use of data (3) 


0% 


0% 


0% 


0% 


0% 


0% 


E. Technology (4) 


0% 


0% 


0% 


0% 


0% 


0% 


Foundational Skills Overall 


CM 


CM 


5% 


CM 


CM 


CM 


Overall 


Average % categorical 
concurrence 


39% 


50% 


48% 


47% 


45% 


30% 



Note: Average percent categorical concurrence by test computed as the average of the percents across organizing 
components. Categorical concurrence is defined as a test having six or more unique test items assessing a 
performance expectation and the percents shown are the percent of performance expectations within an organizing 
component that attained categorical concurrence for each test. Shaded cells indicate the recommended benchmark 
was met. 

Reviewers of the mathematics CCRS found categorical concurrence highest across all 
tests for algebraic reasoning, functions, and numeric reasoning components and lowest 
for the foundational skills. All tests attained categorical concurrence in algebraic 
reasoning, functions and communications and representations. Across all tests, the 
lowest categorical concurrence was observed in probabilistic reasoning, statistical 
reasoning (with one exception), and the cross-disciplinary skills (with the exception of 
the problem solving and— to a lesser extent— reading across the curriculum 
expectations, which were moderately assessed by the tests). 

Although some tests have unique strengths, results are somewhat consistent across 
tests: organizing components low in categorical concurrence tend to be low across all 
tests, organizing components high in categorical concurrence tend to be high across all 
tests. This suggests that, with minor gaps or areas of unique coverage, the placement 
and admissions tests provide similar content coverage relative to the CCRS. 

Table 19 describes the percent of performance expectations within each organizing 
component for which categorical concurrence was established for each 
English/language arts test. For the English/language arts CCRS, categorical 
concurrence is higher for writing than for the foundational skills and reading. 
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It is lowest for research. 



Table 19. Categorical Concurrence by Key Content for English/Language Arts Tests 



Key Content 


Organizing Component 


Percent of Performance Expectations 
that Reached Categorical Concurrence 
for this Test in ELA 


A 


B 


C 


D 


E 


F 


Writing 


I.A. Compose a variety of texts that demonstrate 
clear focus, the logical development of ideas in 
well-organized paragraphs, and the use of 
appropriate language that advances the 
author’s purpose. (5) 


100% 


80% 


100% 


100% 


80% 


80% 


Writing Overall 


100% 


80% 


100% 


100% 


80% 


80% 


Reading 


II. A. Locate explicit textual information and draw 
complex inferences, analyze, and evaluate the 
information within and across texts of varying 
lengths. (11) 


55% 


73% 


46% 


46% 


46% 


46% 


II. B. Understand new vocabulary and concepts 
and use them accurately in reading, speaking, 
and writing. (3) 


0% 


33% 


33% 


33% 


33% 


33% 


II. C. Describe, analyze, and evaluate 
information within and across literary and other 
texts from a variety of cultures and historical 
periods. (4) 


0% 


25% 


0% 


0% 


0% 


0% 


II. D. Explain how literary and other texts evoke 
personal experience and reveal character in 
particular historical circumstances. 


0% 


0% 


0% 


0% 


0% 


0% 


Reading Overall 


14% 


33% 


20% 


20% 


20% 


20% 


Research 


V.B. Select information from a variety of 
sources. (4) 


0% 


0% 


0% 


0% 


0% 


0% 


Research Overall 


0% 


0% 


0% 


0% 


0% 


0% 


Key Cognitive 
Strategies 


A. Intellectual curiosity (2) 


0% 


0% 


0% 


0% 


0% 


0% 


B. Reasoning (4) 


25% 


25% 


0% 


25% 


25% 


25% 


C. Problem solving (3) 


33% 


33% 


33% 


33% 


33% 


33% 


D. Academic behaviors (4) 


25% 


25% 


25% 


50% 


25% 


25% 


E. Work habits (2) 


0% 


0% 


0% 


0% 


0% 


0% 


F. Academic integrity (4) 


0% 


0% 


0% 


0% 


0% 


0% 


Key Cognitive Strategies Overall 


14% 


14% 


10% 


18% 


14% 


14% 


Foundational 

Skills 


A. Reading across the curriculum (8) 


50% 


75% 


63% 


63% 


63% 


63% 


B. Writing across the curriculum (3) 


66% 


66% 


66% 


66% 


66% 


66% 


C. Research across the curriculum (8) 


0% 


0% 


0% 


0% 


0% 


0% 


D. Use of data (3) 


0% 


0% 


0% 


0% 


0% 


0% 


E. Technology (4) 


0% 


0% 


0% 


0% 


0% 


0% 
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Key Content 


Organizing Component 


Percent of Performance Expectations 
that Reached Categorical Concurrence 
for this Test in ELA 


A 


B 


C 


D 


E 


F 


Foundational Skills Overall 


23% 


28% 


26% 


26% 


26% 


26% 


Overall 


Average % categorical concurrence 


30% 


31% 


31% 


33% 


28% 


28% 



Note: Average percent categorical concurrence by test is computed as the average of the percents across organizing 
components. The percents described are the percent of performance expectations within an organizing component 
that attained categorical concurrence. 



Within the key content areas for reading, one organizing component accounts for nearly 
all of the categorical concurrence observed: locate explicit textual information and draw 
complex inferences, analyze, and evaluate the information within and across texts of 
varying lengths. Within the foundational skills, all tests demonstrate categorical 
concurrence on the reading and writing across the curriculum organizing components, 
but no tests demonstrate categorical concurrence with any other organizing component 
within this key content area. 

As with mathematics, the organizing components that are lower in categorical 
concurrence trend lower across all tests, and organizing components higher in 
categorical concurrence trend higher across all tests, indicating that content coverage is 
similar across tests. 

Depth-of- Knowledge Consistency 

To determine the relationship of rigor and cognitive demand between test-items and the 
CCRS, the differences between items and performance-expectation rigor and cognitive- 
demand rating was computed for all matches. Table 20 summarizes the differences 
between the depth of knowledge ratings for test items and the performance 
expectations. 
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Table 20. Average Differences Between Test-Item Rigor and Cognitive Demand and the 
Rigor and Cognitive Demand of the Performance Expectations they Assess 





Matches Between Test 
Items and Performance 
Expectations 


Average Difference 
in Cognitive Demand 

(item cognitive demand 
minus performance-expectation 
cognitive demand) 


Average Difference 
in Rigor 

(item rigor minus performance- 
expectation rigor ) 


Test 


Number 


Mean 


SD 


Mean 


SD 


Mathematics 












A 


4,571 


-0.38 


0.787 


-0.10 


0.263 


B 


5,965 


-0.18 


0.805 


-0.17 


0.228 


C 


12,592 


-0.25 


0.800 


-0.16 


0.215 


D 


9,543 


-0.36 


0.806 


-0.12 


0.244 


E 


3,409 


-0.16 


0.780 


-0.13 


0.232 


F 


1,622 


-0.44 


0.767 


-0.15 


0.243 


English 












A 


2,720 


-0.54 


0.518 


-0.33 


0.330 


B 


5,676 


-0.73 


0.579 


-0.31 


0.340 


C 


3,566 


-0.82 


0.538 


-0.40 


0.310 


D 


3,497 


-0.64 


0.583 


-0.38 


0.325 


E 


2,882 


-0.67 


0.596 


-0.28 


0.337 


F 


1,348 


-0.56 


0.561 


-0.23 


0.339 



Note: Average differences are the average of the individual differences between test items and performance 
expectations for all matches identified on each test. Negative means indicate that on average, the reviewers rated the 
test items lower in cognitive demand or rigor than the performance expectations they assessed. We note that 
significance tests are not often used in typical alignment study methodology and that were they to be applied, many 
confidence intervals would include zero. 

Although all differences are small, the difference between standards and test items in 
cognitive demand and rigor is relatively larger for ELA tests than for mathematics tests. 
The differences are negative in most cases because the performance expectations are 
overall, more rigorous and cognitively demanding than the test items, as is expected 
due to the range of rigor required by the tests to assess a range of student proficiency. 
The smaller differences indicate closer correspondence between the test items and their 
performance expectations. 

While on average, the standards were more rigorous than the test items, this was not 
the case with all performance expectations. For rigor in mathematics, test items rank 
higher than the CCRS in the following categories: 
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• Numeric reasoning, including 1) number representation and 2) number sense 
and concepts 

• Algebraic reasoning, including 1) manipulating expressions and 2) solving 
equations, inequalities, and systems of equations 

• Measurement reasoning, including 1) measurement involving physical and 
natural attributes and 2) systems of measurement 

• Functions, including 1) recognition and representation of functions and 2) 
analysis of functions 

• Communication and representation, including language, terms, and symbols of 
mathematics 

For ELA, the only performance expectation that is rated lower in rigor than assessment 
items is writing. 

In cognitive demand for both mathematics and ELA, the CCRS are more challenging on 
average than are the test items. The exceptions to this, where test items are more 
challenging, are: 

• Numeric reasoning, including 1) number representation, 2) number operations 

• Algebraic reasoning, including 1) manipulating expressions, 2) solving equations, 
inequalities, and systems of equations, and 3) representations 

• Measurement reasoning, including 1) measurement involving physical and 
natural attributes, 2) systems of measurement and 3) measurement involving 
geometry and algebra 

• Probabilistic reasoning, including 1) counting principles and 2) computation and 
interpretation of probabilities 

• Functions, including recognition and representation of functions 

Overall, then, the CCRS are more rigorous and demanding than the corresponding test 
items, except specific mathematics performance expectations as indicated above. 

Range of Knowledge 

In addition to categorical concurrence, the alignment between the CCRS and the 
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assessments are described using range of knowledge. The range of knowledge 
correspondence criterion examines the alignment of assessment items to expectations 
within the CCRS. It describes how a breadth of knowledge the standards expect of 
students corresponds to knowledge needed to correctly answer test items. Range of 
knowledge is calculated as the percent of organizing components that have at least one 
item assessing each performance expectation. This is summarized across reviewers, 
and the suggested benchmark is for a test to have one or more items assess at least 
half of the performance expectations within an organizing component. 

Table 21 describes the percent of performance expectations that are assessed by at 
least one mathematics test item (summed across all reviewers). Recall that each test 
item assesses more than one performance expectation. On average, across all key 
content, the mathematics tests covered between 58 to 81 percent of the content 
described by the CCRS. 



Table 21. Range of Knowledge (Using Total Number of Matches) by Mathematics Test 



Key Content 


Organizing Component 


Percent of Mathematics Performance Expectations 
Assessed By At Least One Test Item 


A 


B 


C 


D 


E 


F 


Numeric 

Reasoning 


A. Number representation 


50% 


50% 


100% 


100% 


100% 


50% 


B. Number operations 


100% 


100% 


100% 


100% 


100% 


100% 


C. Number sense and 
number concepts 


100% 


100% 


100% 


100% 


100% 


100% 


Numeric reasoning Overall 


83% 


83% 


100% 


100% 


100% 


83% 


Algebraic 

Reasoning 


A. Expressions and 
equations 


100% 


100% 


100% 


100% 


100% 


100% 


B. Manipulating expression 


100% 


100% 


100% 


100% 


100% 


100% 


C. Solving equations, 
inequalities, and systems 
of equations 


100% 


50% 


100% 


100% 


100% 


50% 


D. Representations 


100% 


100% 


100% 


100% 


100% 


100% 


Algebraic Reasoning 
Overall 


100% 


88% 


100% 


100% 


100% 


88% 


Geometric 

Reasoning 


A. Figures and their 
properties 


100% 


100% 


100% 


100% 


100% 


100% 


B. Transformations and 
symmetry 


0% 


66% 


33% 


33% 


100% 


33% 
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Key Content 


Organizing Component 


Percent of Mathematics Performance Expectations 
Assessed By At Least One Test Item 


A 


B 


C 


D 


E 


F 


C. Connections between 
geometry and other 
mathematical content strands 


66% 


66% 


66% 


66% 


100% 


66% 


D. Logic and reasoning in 
geometry 


0% 


100% 


100% 


100% 


100% 


100% 


Geometric Reasoning 
Overall 


42% 


83% 


75% 


75% 


100% 


75% 


Measurement 

Reasoning 


A. Measurement involving 
physical and natural 
attributes 


100% 


0% 


66% 


100% 


100% 


100% 


B. Systems of measurement 


0% 


50% 


100% 


100% 


100% 


100% 


C. Measurement involving 
geometry and algebra 


66% 


66% 


67% 


100% 


100% 


100% 


D. Measurement involving 
statistics and probability 


0% 


0% 


0% 


0% 


50% 


50% 


Measurement Reasoning 
Overall 


42% 


29% 


58% 


75% 


88% 


88% 


Probabilistic 

Reasoning 


A. Counting principles 


0% 


0% 


100% 


100% 


100% 


0% 


B. Computation and 
interpretation of probabilities 


0% 


0% 


0% 


50% 


100% 


0% 


Probabilistic Reasoning 
Overall 


0% 


0% 


50% 


75% 


100% 


0% 


Statistical 

Reasoning 


A. Data collection 


0% 


0% 


0% 


0% 


0% 


0% 


B. Describe data 


0% 


25% 


50% 


75% 


75% 


100% 


C. Read, analyze, interpret, 
and draw conclusions 
from data 


0% 


0% 


50% 


50% 


100% 


100% 


Statistical Reasoning 
Overall 


0% 


13% 


33% 


42% 


58% 


67% 


Functions 


A. Recognition and 
representation of functions 


50% 


0% 


50% 


100% 


50% 


50% 


B. Analysis of functions 


100% 


100% 


100% 


100% 


100% 


100% 


C. Model real world situations 
with functions 


50% 


100% 


100% 


100% 


100% 




Functions Overall 


67% 


67% 


83% 


100% 


83% 


75% 


Problem Solving 
and Reasoning 


A. Mathematical problem 
solving 


80% 


80% 


80% 


100% 


80% 


80% 


B. Logical reasoning 


0% 


50% 


100% 


100% 


100% 


100% 


C. Real world problem 
solving 


33% 


100% 


33% 


100% 


100% 


100% 


Problem Solving 
and reasoning Overall 


38% 


77% 


71% 


100% 


93% 


93% 
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Key Content 


Organizing Component 


Percent of Mathematics Performance Expectations 
Assessed By At Least One Test Item 


A 


B 


C 


D 


E 


F 


Communication 

and 

Representation 


A. Language, terms, and 
symbols of mathematics 


66% 


100% 


100% 


100% 


100% 


100% 


B. Interpretation of 
mathematical work 


50% 


100% 


50% 


100% 


100% 


100% 


C. Presentation and 
representation of 
mathematical work 


0 % 


0% 


33% 


33% 


33% 


100% 


Communications and 
Representations Overall 


39% 


67% 


61% 


78% 


78% 


100% 


Connections 


A. Connections among the 
strands of mathematics 


50% 


100% 


50% 


100% 


50% 


100% 


B. Connections of 
mathematics to nature, real- 
world situations, and 
everyday life 


33% 


66% 


33% 


33% 


33% 


100% 


Connections Overall 


42% 


83% 


42% 


67% 


42% 


100% 


Key Cognitive 
Strategies 


A. Intellectual curiosity 


0% 


0% 


50% 


0% 


0% 


0% 


B. Reasoning 


0% 


0% 


0% 


25% 


0% 


25% 


C. Problem solving 


66% 


66% 


67% 


67% 


67% 


100% 


D. Academic behaviors 


0% 


0% 


0% 


0% 


25% 


0% 


E. Work habits 


0% 


0% 


0% 


0% 


0% 


0% 


F. Academic integrity 


0% 


0% 


0% 


0% 


0% 


0% 


Key Cognitive Strategies 
Overall 


11% 


11% 


20% 


15% 


15% 


21% 


Foundational Skills 


A. Reading across the 
curriculum 


12% 


12% 


38% 


25% 


25% 


25% 


B. Writing across the 
curriculum 


0% 


0% 


0% 


0% 


0% 


0% 


C. Research across the 
curriculum 


0% 


0% 


0% 


0% 


0% 


0% 


D. Use of data 


0% 


0% 


0% 


0% 


33% 


100% 


E. Technology 


0% 


0% 


0% 


0% 


0% 


0% 


Foundational Skills Overall 


2% 


2% 


8% 


5% 


12% 


25% 


Overall 


Total 


39% 


50% 


58% 


69% 


72% 


68% 



Note: Percents were computed using the total number of matches (if different reviewers identified the same item as 
assessing a performance expectation , that item counted each time it was identified). Shaded cells indicate the 
recommended benchmark of 50 percent was met. 



Using the criteria that a test have at least one item assessing each performance 
expectation, all the tests sufficiently assess numeric reasoning, algebraic reasoning, 
geometric reasoning, measurement reasoning, functions, problem solving and 
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reasoning, and communication and representation. 

Less well covered are the cross-disciplinary skills. None of the tests sufficiently assess 
this, although all tests do asses the problem solving component of the key cognitive 
strategies, and a single test assesses the use-of-data component of the foundational 
skills and has the best overall assessment of foundational skills. 



The ELA content is less consistently assessed by the tests than the mathematics 
content. Table 22 describes the range of knowledge for the ELA. On average, across all 
key content, the tests covered between 34 to 62 percent of the content described by the 
OCRS. 



Table 22. Range of Knowledge (using total number of matches) by ELA Test 



Key Content 


Organizing Component 


Percent of ELA Performance Expectations 
Assessed by at Least One Test Item 






A 


B 


C 


D 


E 


F 


Writing 


A. Compose a variety of texts that 
demonstrate clear focus, the logical 
development of ideas in well-organized 
paragraphs, and the use of appropriate 
language that advances the author's 
purpose. 


100% 


100% 


100% 


100% 


100% 


100% 




Writing Overall 


100% 


100% 


100% 


100% 


100% 


100% 




A. Locate explicit textual information and 
draw complex inferences, analyze, and 
evaluate the information within and across 
texts of varying lengths. 


73% 


73% 


55% 


45% 


64% 


64% 


Reading 


B. Understand new vocabulary and 
concepts and use them accurately in 
reading, speaking, and writing. 


0% 


33% 


33% 


33% 


33% 


33% 


C. Describe, analyze, and evaluate 
information within and across literary and 
other texts from a variety of cultures and 
historical periods. 


0% 


50% 


25% 


0% 


0% 


0% 




D. Explain how literary and other texts evoke 
personal experience and reveal character in 
particular historical circumstances. 


0% 


50% 


0% 


0% 


0% 


0% 




Reading Overall 


18% 


52% 


28% 


20% 


24% 


24% 


Research 


A. Formulate topic and questions. 


0 


0 


0 


0 


0 


0 
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Key Content 


Organizing Component 


Percent of ELA Performance Expectations 
Assessed by at Least One Test Item 


A 


B 


C 


D 


E 


F 


B. Select information from a variety of 
sources. 


0% 


0% 


0% 


0% 


75% 


0% 


Research Overall 


0% 


0% 


0% 


0% 


38% 


0% 


Key Cognitive 
Strategies 


A. Intellectual curiosity 


0% 


0% 


0% 


100% 


100% 


0% 


B. Reasoning 


100% 


75% 


25% 


100% 


100% 


75% 


C. Problem solving 


67% 


33% 


33% 


100% 


67% 


33% 


D. Academic behaviors 


25% 


25% 


25% 


100% 


50% 


25% 


E. Work habits 


0% 


0% 


0% 


100% 


0% 


0% 


F. Academic integrity 


25% 


0% 


0% 


100% 


50% 


25% 


Key Cognitive Strategies Overall 


36% 


22% 


14% 


100% 


61% 


26% 


Foundational 

Skills 


A. Reading across the curriculum 


63% 


88% 


75% 


75% 


75% 


75% 


B. Writing across the curriculum 


67% 


100% 


67% 


100% 


100% 


67% 


C. Research across the curriculum 


0% 


0% 


0% 


100% 


0% 


0% 


D. Use of data 


0% 


0% 


0% 


100% 


0% 


33% 


E. Technology 


0% 


0% 


0% 


75% 


0% 


0% 


Foundational Skills Overall 


26% 


38% 


28% 


90% 


35% 


35% 


Overall 


Total 


36% 


42% 


34% 


62% 


52% 


37% 



Note: Percents were computed using the total number of matches (so that if different reviewers identified the same 
item as assessing a performance expectation, that item counted each time it was identified). Shaded cells in the table 
indicate the recommended benchmark of 50 percent was met. 



While categorical concurrence identifies the tests that have enough items to reliably 
assess individual performance expectations, range of knowledge identifies the tests that 
assess the most performance expectations. Using the criteria that tests have at least 
one item assessing each performance expectation, all tests sufficiently assess writing 
only. Reading and the cross-disciplinary skills are less well covered by the tests. As with 
mathematics, the organizing components high in the range of knowledge tended to be 
high across all tests, regardless of the number of items. When these components were 
low, they tended to be low across all tests. 

Interpretation of Findings 

Results suggest that most of the admissions and placement tests reviewed align to a 
moderate to high level with the CCRS performance expectations in mathematics and 
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ELA. None of the tests aligns at a high level with the cross-disciplinary skills. The tests 
exhibited similar results for categorical-concurrence and range-of-knowledge measures. 

In mathematics, alignment (as measured by both categorical concurrence and range of 
knowledge) is highest for numeric reasoning, algebraic reasoning, functions, problem 
solving and reasoning, and communications and representations; lower for connections, 
geometric reasoning, measurement reasoning, and statistical reasoning; and lowest for 
foundational skills, probabilistic reasoning, and key cognitive strategies. These findings 
are consistent with the tests specifications. All of the tests are designed to assess 
numeric reasoning, algebraic reasoning, geometric reasoning, and functions; only one- 
third are designed to assess measurement reasoning, probabilistic reasoning, statistical 
reasoning, and problem solving reasoning; and none of the tests specify the 
connections content that are in the CCRS. 

In ELA, alignment (as measured by categorical concurrence and range of knowledge) is 
highest for writing, lower for reading and foundational skills, and lowest for key cognitive 
strategies. Although minimally assessed by one test, none of the tests provide sufficient 
assessment of the research performance expectations contained in the CCRS. These 
findings are somewhat unexpected because all ELA tests are designed to assess 
reading and writing, yet only writing shows consistent and sufficient alignment. 

Rater agreement was higher for mathematics items and standards than for ELA items 
and standards, suggesting perhaps that college readiness standards are still a new 
phenomenon for reviewers and that special attention needs to be paid to developing 
stronger common mental maps of the standards on the part of all reviewers before they 
begin rating test items. Reviewer agreement was higher for cognitive-demand ratings 
than for rigor ratings, which is counterintuitive as rigor is rated on a smaller scale (1-3 
rather than 1-4) and evaluating whether an item or a standard is at the level of an entry- 
level college student might appear to be a more straightforward distinction to make than 
determining the thought process required by an item or standard. Other researchers 
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report similar findings (Herman, Webb, Zuniga, 2005), suggesting that it may be easier 
to apply cognitive-demand ratings consistently than it is to apply rigor ratings 
consistently. While an agreement of 0.8 or higher is ideal, other studies in this area have 
yielded typical agreement measures for alignment studies closer to 0.5 (Blank, 2007), a 
criterion level that this study exceeds. 

Analysis of the rigor and cognitive-demand ratings shows similar patterns for both 
measures. For the CCRS subject areas, the ELA performance expectations are more 
rigorous and cognitively demanding than are the cross-disciplinary skills and the 
mathematics standards. The mathematics standards are the least rigorous, averaging 
below the level at which an entry-level college student should be expected to perform. 

The ELA standards are the highest and averaged close to the level at which entry-level 
college students should be expected to perform. The cross-disciplinary skills fall 
between mathematics and ELA. 

The most rigorous and cognitively demanding mathematics concepts involve statistical 
and logical reasoning. Although statistics is increasingly taught in high school, its 
concepts may be more abstract than typical high school mathematics content. Logical 
reasoning is a complex skill that is difficult to assess with multiple-choice items. The 
CCRS are more rigorous and cognitively demanding than the test items in this area. 

This should not be unexpected because the CCRS describe the knowledge and skills 
that college-ready students should have, while the test items assess a much broader 
range of proficiency in order to determine the level necessary to enter a credit-bearing 
course. 

Some CCRS are not assessed by any items on any test. For mathematics, these 
include skills that may not be assessable by multiple-choice test items, including 
practical mathematics skills (planning a study, collecting data, analyzing relationships 
between paired data, recognizing reliability of statistical results). For ELA, these include 
practical reading skills (using reference guides to confirm the meanings of new words or 
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concepts, reading a variety of texts from world literatures, etc.) and practical research 
skills (formulating a research question, exploring a research topic, using source material 
ethically, etc.) and include skills not practical in a typical standardized testing 
environment, even with the use of constructed-response type items. 

The lack of coverage for these standards suggests that to assess the full range of 
knowledge and skills necessary to be college ready in the state of Texas may eventually 
require additional methods. Previous research supports this, suggesting that item type is 
an important factor in determining what assessable content is (Rothman, 2004), and 
that using multiple methods, including performance assessment and constructed 
response-type items in more subjects than writing, may be necessary to cover fully 
content that is not assessable through multiple-choice items. It may be worth 
considering combining multiple-choice test results with end-of-course-exam results, 
senior demonstrations or projects, performance assessments, or portfolios in order to 
gauge student readiness across the full set of college and career standards. 

Additional work may be needed to standardize the methodology for including test pools 
in alignment studies. As noted earlier, previous researchers using both test forms and 
item pools in an alignment study have tended not to address explicitly the likely impact 
on findings resulting from the different number of items. Although similar results were 
found across test pools and forms, metrics that rely upon counts may not always be 
directly comparable. Future research should be conducted to refine methods for direct 
comparisons of test forms to test-item banks. 

Finally, based on this study, the recommendation is to consider the development and 
introduction of measures and assessments that are based on broader and more robust 
conceptions of college readiness that are consistent with the full set of Texas College 
and Career Readiness Standards and a fuller set of college readiness expectations. 

Such an approach would help lead the nation in a direction in which college readiness 
became the expression of a full range of skills and capabilities, some of which can be 
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measured by current commercially available instruments and others that will require 
new tools and methods. Such a system would be able to provide the full range of 
information that is necessary to know if students are truly college ready and to identify 
and diagnose the specific areas where students need additional help, practice, support, 
and skill building to be prepared to succeed in entry-level college courses. 
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Appendix A: Texas College and Career Readiness Cross- 
Disciplinary Standards 

The following cross-disciplinary standards are excerpted from the Texas College and 
Career Readiness Standards (CCRS). To view all CCRS, including the 
English/language arts, mathematics, science, and social standards, please go to: 
http://www.thecb.state.tx.us/collegereadiness/TCRS.cfm 

The cross-disciplinary standards are organized into two major areas: Key Cognitive 
Skills and Foundational Skills. The Key Cognitive Skills specify intellectual behaviors 
that are prevalent in entry-level college courses. The list includes intellectual curiosity, 
reasoning, problem solving, academic behaviors, work habits, and academic integrity. 
Foundational Skills consist of proficiencies students need to be able to transfer 
knowledge and apply it across the curriculum. These include reading, writing, 
conducting research, understanding and using data, and using technology. 

The first three levels of the cross-disciplinary standards are written to apply across 
subject areas. The performance indicators found in the appendix illustrate how the 
cross-disciplinary standards are manifested within the subject areas. The Vertical 
Teams created an example in each subject area of at least one performance indicator 
that could be applied in that subject area. These indicators are meant to exemplify how 
the cross-disciplinary standards could be demonstrated in all subject areas. 

I. Key Cognitive Skills 

A. Intellectual curiosity 

1. Engage in scholarly inquiry and dialogue. 

a. Identify what is known, not known, and what one wants to know in a 
problem. 

b. Conduct investigations and observations. 

c. Cite examples or illustrations in which a clear-cut answer cannot be 
reached. 

2. Accept constructive criticism and revise personal views when valid 
evidence warrants. 

a. Articulate a point of view and provide valid evidence to support findings. 

b. Demonstrate willingness to take intellectual risks by investigating novel, 
controversial, or unpopular opinions or conclusions. 

c. Examine alternative points of view, taking different roles to defend, oppose, 
and remain neutral on issues. 

d. Recognize conflicting information or unexplained phenomena. 
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B. Reasoning 

1. Consider arguments and conclusions of self and others. 

a. Know and apply logic to analyze patterns and descriptions and to evaluate 
conclusions. 

b. Cite valid examples or illustrations that support the conclusions. 

c. Question whether the claims and conclusions of self and others are 
supported by evidence. 

d. Identify counter examples to disprove a conclusion. 

2. Construct well-reasoned arguments to explain phenomena, validate 

conjectures, or support positions. 

a. Participate in a debate that is based on facts and has a logical structure. 

b. Construct a visual presentation, including hypothesis, data, results, and 
conclusion. 

c. Write a paper that addresses counter-arguments to advocated positions. 

d. Recognize and apply techniques of statistical or probabilistic analysis to 
judge reliability of information. 

e. Organize an argument separating fact from opinion. 

3. Gather evidence to support arguments, findings, or lines of reasoning. 

a. Use different kinds of data (e.g., case studies, statistics, surveys, 
documents) to support an argument. 

b. Evaluate evidence in terms of quality and quantity. 

c. Describe limitations of data collection methods. 

4. Support or modify claims based on the results of an inquiry. 

a. Refine claims and adjust a position in response to inquiry. 

b. Review and check strategies and calculations, using alternative approaches 
when possible. 

C. Problem solving 

1. Analyze a situation to identify a problem to be solved. 

a. Represent and/or restate the problem in one or more ways (e.g., graph, 
table, equation), showing recognition of important details and significant 
parameters. 

b. Break complex problems into component parts that can be analyzed and 
solved separately. 

c. Apply previously learned knowledge to new situations. 

d. Analyze a media report, identify any misuse of statistics, and suggest ways 
to more accurately depict this information. 

2. Develop and apply multiple strategies to solve a problem. 

a. Use a range of standard methods, devices, techniques, and strategies to 
gather and analyze information. 

b. Use knowledge gained from other subject areas to solve a given problem. 

3. Collect evidence and data systematically and directly relate to solving a 

problem. 
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a. Use general and specialized reference works and databases to locate 
sources. 

b. Collect evidence and data directly related to solving the problem and 
eliminate irrelevant information. 

c. Produce charts, graphs, and diagrams accurately, including scale, labeling, 
units, and organization. 

d. Present the collected data visually, describe the data collection procedure, 
and defend choosing that procedure over other possibilities. 

D. Academic behaviors 

1. Self-monitor learning needs and seek assistance when needed. 

a. Ask questions to check for understanding or to clarify information. 

b. Use a systematic method for recording, storing, and organizing materials and 
resources; avoid haphazard or messy accumulation of information. 

2. Use study habits necessary to manage academic pursuits and 

requirements. 

a. Manage time effectively to complete tasks on time. 

b. Demonstrate accurate note-taking. 

c. Use the appropriate level of detail necessary to complete an assigned task. 

d. Balance academic and non-academic activities to successfully participate in 
both. 

3. Strive for accuracy and precision. 

a. Collect and report experimental data carefully and correctly. 

b. Produce charts, graphs, and diagrams accurately, including scale, labeling, 
units, and organization. 

c. Eliminate irrelevant information from an assignment. 

4. Persevere to complete and master tasks. 

a. Persevere until a task is completed by working even when faced with 
uncertainty or open-ended assignments. 

b. Seek assistance when needed to complete the assignment. 

c. Recognize when a task is completed. 

E. Work habits 

1. Work independently. 

a. Plan a project, establish its parameters, and complete it with minimal 
supervision, seeking assistance accordingly. 

b. Follow directions or procedures independently. 

c. Complete assignments outside the classroom setting in a timely manner. 

2. Work collaboratively. 

a. Work collaboratively with students from various cultural and ethnic 
backgrounds. 

b. Distinguish between situations where collaborative work is appropriate and 
where it is not. 
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c. Work in small groups to investigate a problem or conduct an experiment. 

F. Academic integrity 

1. Attribute ideas and information to source materials and people. 

a. Document the work of others, giving credit where credit is due and never 
claim credit for work that is not one’s own. 

b. Use standard bibliographic and reference citation formats, choosing the 
style appropriate to the subject and the audience. 

c. Define plagiarism and articulate the consequences of academic dishonesty. 

2. Evaluate sources for quality of content, validity, credibility, and relevance. 

a. Verify validity of a source within a submitted work. 

b. Compare and contrast coverage of a single topic from multiple media 
sources. 

3. Include the ideas of others and the complexities of the debate, issue, or 
problem. 

a. Present multiple perspectives of an issue. 

b. Represent accurately the data, conclusions, or opinions of others. 

4. Understand and adhere to ethical codes of conduct. 

a. Follow copyright laws and restrictions. 

b. Use technology responsibly (e.g., avoiding malice, misrepresentation, or 
misleading use of information). 

II. Foundational Skills 

A. Reading across the curriculum 

1. Use effective prereading strategies. 

a. Use the title, knowledge of the author, and place of publication to make 
predictions about a text. 

b. Use a table of contents to preview a text and understand its design. 

c. Scan headline sections or other division markers, graphics, or sidebars to 
form an overview of a text. 

2. Use a variety of strategies to understand the meanings of new words. 

a. Use context clues, including definitions, examples, comparison, contrast, 
cause and effect, and details provided in surrounding text. 

b. Consult references (e.g., dictionary, thesaurus) effectively. 

c. Understand notation specific to discipline (e.g., mathematical notation, 
scientific symbols). 

3. Identify the intended purpose and audience of the text. 

a. Predict purpose and audience of a text based on the title, preface, and other 
features of a text. 

b. Explain how the language of an effective text targets an intended audience. 

c. Explain the importance of a technical and/or scientific article. 
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4. Identify the key information and supporting details. 

a. Outline a chapter of an informational text. 

b. Summarize the major points in a text, and use graphic organizers (e.g., 
concept maps, diagrams) to organize ideas and concepts in a visual 
manner. 

c. Analyze connections between major and minor ideas. 

d. Identify and define key terminology from technical and/or scientific 
documents. 

5. Analyze textual information critically. 

a. Identify faulty premises in an argument. 

b. Identify stated and implied assumptions. 

c. Identify conclusions unsupported by sufficient evidence in informational 
texts. 

d. Use inductive and deductive reasoning. 

e. Draw conclusions based on evidence, support, or data through logical 
reasoning. 

f. Compare a primary source and an interpretation in a textbook. 

6. Annotate, summarize, paraphrase, and outline texts when appropriate. 

a. Outline an informational or literary text. 

b. Annotate text for comprehension and analysis. 

c. Summarize an article to demonstrate comprehension. 

d. Paraphrase a writer’s ideas or findings. 

7. Adapt reading strategies according to structure of texts. 

a. Identify a variety of textual forms and genres (e.g., long and short texts) and 
adapt reading strategies accordingly. 

b. List strategies to use during reading, including: 

• Anticipate and predict what information the text is likely to contain. 

• Monitor understanding by self-questioning. 

• Use strategies (e.g., mental imagery, paraphrasing, information in 
glossaries) to re-examine the text if comprehension fails. 

• Reread difficult passages. 

• Read ahead for additional clarification. 

• Seek assistance for clarification. 

• Self-monitor and summarize the information gained. 

c. Explain how form or genre communicates meaning. 

8. Connect reading to historical and current events and personal interest, 
a. Locate an article or source that relates to a class topic and explain the 

relevance. 

B. Writing across the curriculum 

1. Write clearly and coherently using standard writing conventions. 

a. Prepare a topic proposal that specifies a purpose and justifies the choice of 
audience to achieve that purpose. 
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b. Craft a thesis statement that articulates a position and list relevant evidence 
and examples in logical groupings. 

c. Use symbols, diagrams, graphs, and words to communicate ideas. 

d. Use appropriate terminology and data expression to communicate 
information in a concise manner. 

e. Use a variety of reference guides for citation conventions, grammar, 
mechanics, and punctuation. 

2. Write in a variety of forms for various audiences and purposes. 

a. Present an argument supported by relevant evidence, examples, and 
counterarguments. 

b. Prepare a summary or abstract of a journal article or report, extracting in brief 
form the pertinent information. 

c. Evaluate articles by analyzing the study design, data source, graphical 
representation of data, and analyzed data results reported (or not reported). 

d. Write a reflection about the process selected to conduct research or solve a 
problem. 

e. Write accurate and understandable lab reports and technical documents. 

3. Compose and revise drafts. 

a. Submit a writing assignment to be proofread by a teacher, parent, or other 
student. Revise the paper, incorporating constructive criticism when 
appropriate. 

b. Edit text for correct spelling, capitalization, and punctuation. 

c. Edit for appropriate tense and voice. 

d. Edit for correct word use. 

e. Use a variety of reference guides for citation conventions, grammar, 
mechanics, and punctuation. 

f. Submit a final draft that is easily read and has few or no grammatical or 
spelling errors. 

C. Research across the curriculum 

1 . Understand which topics or questions are to be investigated. 

a. Formulate research questions. 

b. Use strategies like those in the writing process to generate questions and 
areas to pursue. 

c. Consult previous studies or conduct interviews with experts to identify 
questions central to a research topic. 

d. Propose explicit, testable hypotheses, using the “if ..., then ...” format. 

2. Explore a research topic. 

a. Produce an annotated list of sources consulted, differentiating among 
primary, secondary, and other sources and explain their relevance to the 
research topic. 

b. Outline the most significant controversies or questions on a research topic. 

c. Plan an investigative study. 
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d. Explain reasons for valid competing points of view on a given topic. 

3. Refine research topic based on preliminary research and devise a timeline 

for completing work. 

a. Gather information from a variety of relevant sources. 

b. Use general and specialized reference works and databases to locate 
sources. 

c. Locate electronic sources, when appropriate, using advanced search 
strategies. 

d. Select an appropriate range of source materials. 

e. Analyze a wide range of sources, including technical texts, primary and 
secondary sources, conflicting points of view, and interdisciplinary research 
when appropriate. 

f. Design and carry out hands-on experimental investigations, choosing 
appropriate apparatuses, identifying controls and variables, tentatively 
predicting the outcome of the procedures, and evaluating whether actual 
results agree with predicted results. 

g. Use numerical and mathematical tools such as software, including 
databases, spreadsheets, and other tools, in investigations and 
explanations. 

4. Evaluate the validity and reliability of sources. 

a. State explicitly characteristics or identifying features that indicate accuracy 
or reliability of sources, to determine whether sources are biased, 
incomplete, or otherwise unreliable. 

b. Follow a set of criteria to determine the validity and reliability of sources. 

c. Identify claims found in one or more of the sources that require support or 
verification, and evaluate the information’s validity. 

d. Evaluate the data presented in graphics, tables, charts, and maps when 
appropriate to the topic. 

5. Synthesize and organize information effectively. 

a. Select quotations and evidence that support the thesis. 

b. Determine what evidence best supports conclusions. 

c. Use well-organized strategies to collect and organize information gathered. 

d. Determine the best order for presenting evidence that supports conclusions. 

6. Design and present an effective product. 

a. Determine the best order for presenting major and minor points. 

b. Design a report using features such as headings and graphics appropriate 
to the writing task. 

c. Use a citation system specified by or appropriate to the assignment. 

7. Integrate source material. 

a. Integrate source material into text by a combination of accurately 
summarizing, paraphrasing, and quoting. 

b. Balance use of source material with relevant explanations. 

c. Use source material ethically. 
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d. Understand and avoid all types of plagiarism. 

8. Present final product. 

a. Use appropriate media for presentation of research results. 

b. Document sources using a standard format appropriate to the subject area. 

D. Use of data 

1. Identify patterns or departures from patterns among data. 

a. Identify patterns from multiple representations of data such as graphical and 
tabular forms. 

b. Review current news events and evaluate possible connections (e.g., linking 
economic data with political events). 

2. Use statistical and probabilistic skills necessary for planning an 

investigation and collecting, analyzing, and interpreting data. 

a. Create representations of data (e.g., data tables, correctly labeled and 
scaled graphs, narrative descriptions). 

b. Evaluate a given published report for missing information and misuse of 
data. 

3. Present analyzed data and communicate findings in a variety of formats. 

a. Compose a written document detailing a research project. 

b. Use appropriate visuals and statistical results to convey findings to a 
specified audience. 

E. Technology 

1 . Use technology to gather information. 

a. Use the Internet or other appropriate technologies to post survey questions 
on an assigned topic. 

b. Use devices to measure physical properties. 

c. Use online databases to access scholarly work on an assigned research 
topic. 

2. Use technology to organize, manage, and analyze information. 

a. Use data analysis software to analyze survey results. 

b. Use spreadsheets to manage and organize statistical data. 

3. Use technology to communicate and display findings in a clear and 

coherent manner. 

a. Create spreadsheets and graphs to communicate findings in a presentation 
that includes graphics, visuals, or other supporting images. 

b. Utilize technology to present information and/or data in a variety of ways. 

4. Use technology appropriately. 

a. Explain how technology is a useful and effective tool to communicate 
findings. 

b. Identify when technology may not be necessary or appropriate to 
communicate findings. 

c. Formulate strategies to communicate findings with and without technology. 
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Appendix B: Scale Items & Rationale Statements 
for Examining the Alignment between the Texas College 
and Career Readiness Standards and Entry-Level College 
Courses at Texas Postsecondary Institutions 

Most Necessary for Preparation to Succeed in this Course 

• This element is critical for success in the course 

• Course is taught with the assumption that students already know this information 

• This element will not be retaught in this course 

• Students will have difficulty succeeding in the course if they have not learned this 
element previously 

• Other - provide reason 

More Necessary for Preparation to Succeed in this Course 

• This element is important for success in the course 

• Course is taught with the assumption that students are at least familiar with or 
aware of this element 

• This element will be reviewed only and not retaught in this course 

• Students will benefit from having learned this element previous to the course but 
can probably relearn it during the course and still succeed in the course 

• Other - provide reason 

Less Necessary for Preparation to Succeed in this Course 

• Student knowledge of and familiarity with this element may be helpful 

• Course is taught with the assumption that students are familiar with the element 
on a very general level 

• This element will be taught in some detail in the class 

• Even if students have not learned this previously, they will be able to learn it in 
the course at a level sufficient to succeed in the course 

• Other - provide reason 

Least Necessary for Preparation to Succeed in this Course 

• Students need only minimal knowledge of and familiarity with this element 

• Course is taught with the assumption that students may be only vaguely aware of 
the element 

• This element will be taught as new material in this course even if students have been 
taught it before 

• Students will be able to succeed in this course even if they only have a very general 
awareness or understanding of this element when they enter the course 

• Other - provide reason 
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Not Necessary for Preparation to Succeed in this Course 

• This element is too advanced for this course 

• This element will be encountered for the first time in subsequent courses in the 
subject area 

• This element is too specialized or specific for this course 

• This element is irrelevant to this course 

• This element will be introduced as new material in this course with the assumption 
that students have not learned anything about it before this course 

• Other - provide reason 
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Appendix C: Reference Course Profile: 

English Composition II 

ENGL 1302 

Principles and techniques of written, expository, and persuasive composition; analysis 
of literary, expository, and persuasive texts; and critical thinking.* 

Reference Course Profile: Overview and Purpose 

A Reference Course Profile represents current practice in entry-level college courses in 
Texas. Twenty-six course profiles were developed from an extensive study of over 900 
entry-level general education and CTE college courses offered at Texas institutions of 
higher education. Each course profile consists of a course description, sample syllabus, 
reference lists, attendant course materials (such as assignments, assessments, and, in some 
cases, scoring rubrics), and the Texas College and Career Readiness Standards addressed 
within the course. 

The Profiles provide a reference point for college readiness that fosters increased 
transparency between secondary and postsecondary education. They are not intended to 
be mandatory, prescriptive, or best practice. 



Prereq u isites and Prior Knowled g e 

Most college-level syllabi do not list all prior knowledge required to succeed in the 
course; it is usually an underlying assumption. For the purpose of this Reference Course 
Profile, the required prior knowledge and skills students need to be successful in the 
course are explicitly stated to help both secondary and postsecondary faculty in 
establishing goals and expectations for their students. The knowledge and skills reflected 
in the outline in this section are pulled directly from the Texas College and Career 
Readiness Standards (CCRS), written and validated by Texas faculty in 2007-8. The CCRS 
are available online at: http://www.thecb.state.tx.us/collegereadiness/CRS.pdf 



Prior to enrolling in this course, students must 

• Satisfy Texas Success Initiative (TSI) requirements set by the institution as 
described in Coordinating Board rule (Texas Administrative Code, Chapter 4, 
Subchapter C). 

• ENGL 1301 or its equivalent. 



From the course description appearing in the Lower Division Academic Course Guide Manual (ACGM) 
http:/ /www. thecb. state, tx. us/AAR/UndergraduateEd/WorkforceEd/acgm. htm 
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In addition, students should have the following College and Career Readiness 
Standards skills. Only the specific standards and performance expectations pertinent to 
the course are listed on the following pages. 

English/Language Arts College and Career Readiness Standards 

I. Writing 

B. Compose a variety of texts that demonstrate clear focus, the logical 
development of ideas in well-organized paragraphs, and the use of 
appropriate language that advances the author’s purpose. 

II. Reading 

A. Locate explicit textual information, draw complex inferences, and analyze and 
evaluate the information within and across texts of varying lengths. 

B. Understand new vocabulary and concepts and use them accurately in 
reading, speaking, and writing. 

III. Speaking 

A. Understand the elements of communication both in informal group discussion 
and formal presentations. 

B. Develop effective speaking styles for both group and one-on-one situations. 

IV. Listening 

A. Apply listening skills as an individual and as a member of a group in a variety 
of settings. 

B. Listen effectively in informal and formal situations. 

V. Research 

A. Formulate topic and questions. 

B. Select information from a variety of sources. 

Cross-Disciplinary Standards 

I. Key Cognitive Skills 

A. Intellectual Curiosity 

B. Reasoning 

C. Problem Solving 

D. Academic Behaviors 

E. Work Habits 

F. Academic Integrity 

II. Foundational Skills 

A. Reading Across the Curriculum 

B. Writing Across the Curriculum 

C. Research Across the Curriculum 

D. Use of Data 

E. Technology 
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Course Objectives 

Course objectives include the course-specific skills and knowledge that students will 
possess upon completion of the course. They assist postsecondary faculty in clarifying the 
goals of their courses and provide a clear picture of the expectations students will 
encounter once they begin college. This sample list of objectives was adapted from syllabi 
submitted in 2008 by Texas college faculty. 



Upon completion of the course the student will be able to: 

1 . Demonstrate critical thinking skills as evidenced by the ability to analyze facts, 
synthesize factual information, and evaluate opinions in light of the facts 
presented throughout this course. 

2. Demonstrate the ability to recognize and use both deductive and inductive logic 
and to recognize logical fallacies in one’s own writing and in the writing of others. 

3. Write a well-developed, coherent essay with a minimum of grammatical errors. 

4. Utilize library resources effectively. 

5. Document sources clearly while supporting a thesis with both primary and 
secondary sources. 

6. Demonstrate revising and editing skills. 

7. Demonstrate effective speaking skills. 

Research Objectives: 

This course is a “research” course, which means that it intends to help students grapple 
with the notion of scholarly inquiry. That is, what is research for? What are its 
objectives? What are its methods? What are its shortcomings? Students should learn to 
appreciate research as a highly social matter: that real research must make some kind 
of contribution to the understanding of people besides just the researcher. 

Critical thinking is an important objective of this course. Critical thinking is a process 
involving higher order thinking skills. These skills include, but are not limited to, 
application, analysis, synthesis, and evaluation of factual information. 

A well-cultivated critical thinker 3 : 

• Raises vital questions and problems, formulating them clearly and precisely. 

• Gathers and assesses relevant information, using abstract ideas to interpret it 
effectively. 

• Comes to well-reasoned conclusions and solutions, testing them against relevant 
criteria and standards. 

• Thinks open-mindedly within alternative systems of thought, recognizing and 
assessing their assumptions, implications, and practical consequences. 

• Communicates effectively with others in finding solutions to complex problems. 



3 Paul, Richard & Elder, Linda. The Miniature Guide to Critical Thinking Concepts & Tools. Dillon Beach: The 
Foundation for Critical Thinking, 2005. 
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Reading Objectives: 

Reading at the college level means having the ability to analyze and interpret a variety 
of printed materials: books, articles, and documents. 

Speaking Objectives: 

Effective speaking is the ability to communicate orally in clear, coherent, and persuasive 
language appropriate to purpose, occasion, and audience. 

Computer Literacy Objectives: 

Computer literacy at the college level means having the ability to use computer-based 
technology in communicating, solving problems, and acquiring information. Core- 
educated students should have an understanding of the limits, problems, and 
possibilities associated with the use of technology and should have the tools necessary 
to evaluate and learn new technologies as they become available. 

Sample Textbooks and Materials 

This list is comprised of texts that appear on course syllabi submitted in 2008 by faculty 
teaching entry-level college courses most representative of current practice in Texas. This 
list is not exhaustive, prescriptive, or required. 

• Gibaldi, Joseph. MLA Handbook for Writers of Research Papers. 6 ffi Ed. Modern 
Language Association of America, 2003. 

• Hacker, Diana. A Writer’s Reference, 6 th Ed. Boston: Bedford/St. Martins, 2007. 

• Lunsford, Andrea A., John J. Ruszkiewicz, and Keith Walters. Everything’s an 
Argument (with readings). 4 th Ed. Boston: Bedford/St, Martins, 2007. 

• Kennedy, X. J. and Dana Gioia. Literature: An Introduction to Fiction, Poetry, and 
Drama. 10 th Ed. New York: Longman, 2005. 

• Kress, Anne and Suellyn Winkle. Next Text: Making Connections Across and 
Beyond the Disciplines. Boston: Bedford/St. Martin’s, 2008. 

• Palmquist, Mike. The Bedford Researcher. 2 nd Ed. Boston: Bedford/St. Martin’s, 
2006. 

• Maimon, Elaine P., Janice H. Peritz, and Kathleen Blake Yancey. The New 
McGraw-Hill Handbook. Boston: McGraw-Hill, 2007. 

• Mollick, Kathleen et al. The Popken Writer: A Collection of Student Research 
Writing. Vol. 2. Boston: Houghton-Mifflin, 2008. 

• Rottenberg, Annette T. and Donna Haisty Winchell. Elements of Argument. 9 th Ed. 
Boston: Bedford/St. Martins, 2009. 

• Ruszkiewicz, John, Daniel E. Seward, and Maxine Hairston. SF Writer. 4 th Ed. 
Upper Saddle River, NJ: Prentice Hall, 2007. 

• Troyka, Lynn. Simon and Schuster Handbook for Writers with i-Book. 8 th Ed. 
Prentice Hall. 2006. 

• Wood, Nancy. Perspectives on Argument. 5 th Ed. Upper Saddle River, NJ: 

Prentice Hall. 2006. 
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• A good college-level desk dictionary. 

Sample Methods of Instruction 

Students should be prepared to encounter a variety of instructional methods, as faculty 
indicate the use of several beyond the lecture format. The list of methods of instruction has 
been adapted from course syllabi submitted in 2008 by faculty teaching entry-level 
college courses most representative of current practice in Texas. The approximate 
percentage of time allocated to each instructional method is also indicated. 



1 . Lecture - 20% 

a. Lecture is defined as a method of instruction in which the instructor has full 
responsibility for presenting material orally. 

b. Lectures will take place in the form of informal lectures, in which active 
student participation, such as questioning and answering, will be included. 

2. Full-class Discussion of Readings and Rhetorical Ideas - 30% 

a. Students will be expected to come to class ready to contribute to class 
discussion. 

b. Students will be expected to listen and respond to each other’s comments. 

3. Group Discussion - 1 0% 

a. In addition to class discussion, students will be expected to take turns 
facilitating small group discussions both in and outside of class time. 

4. Peer Review of Writing - 1 0-1 5% 

a. Students will be expected to review the work of classmates and provide and 
accept constructive feedback. 

5. One-on-one Conferences with Instructor - 1 0-1 5% 

a. Conferencing with the instructor on an individual basis allows students to get 
critical feedback of work. 

6. Library Work- 15% 

a. Students are expected to become familiar with library resources and research 
methods. 

Sam p le Assi g nments and Assessments 

A typical number of formal assignments for ENGL 1 302 is four to eight essays. The course 
will also include quizzes, a final exam, and a number of other assignments. Below are the 
kinds of assignments that might be expected and their percentage of the final grade. 



1 . Writing Assignments - 55% 

a. Students will complete a series of assignment culminating in a final research 
paper 

2. Oral Presentation of research - 5% 

a. Students will be required to give a brief oral presentation of their research in 
front of the class. 

3. Final Exam - 20% 
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4. Daily Work - 10% 

a. Daily work will consist of small, in-class assignments, as well as a reading 
journal that students will turn in periodically. 

5. Quizzes - 1 0% 



Sample Schedule 

Samplings of assignments and assessments have been provided in the Reference Course 
Profile materials. Bolded blue text indicates when a document is provided and a link to 
that document is available. The time allocated for students to complete the assignment is 
also indicated. The list of topics, as well as the overall pacing of the course, has been 
adapted from course syllabi submitted in 2008 by faculty teaching entry-level college 
courses most representative of current practice in Texas. 



Week 


Reading/Discussion Topics 


Assignments & Assessments 


1 


Read Wood, Ch. 1 & 3 
Read Brady pp. 59-60 


Assianment 1: Readina Journal 


2 


Read Wood, Ch. 4. pp. 109-113 


Assianment 2: Summary Connect Paper 

(3-4 hours) 


3 


Read Wood, Ch. 5. 


Turn in Reading Journal 


4 


Read “On Women’s Right to Vote” Read 
Wood, “Writing an Annotated Bibliography” pp. 
327-31 


Assianment 3: Research Proposal (2 hours) 


5 


Introduce Research Project 


Turn in Reading Journal 
Quiz 1 


6 


Student/Instructor Conferences 


Assianment 4: Research Paper ( 5-6 hours) 


7 


Research Project Workshop 


Turn in Reading Journal (30-60 minutes) 


8 


Research Presentations 
Wood, Ch. 6 
Read Wood, Ch. 7 


Assessment 1 Reader Response Rubric provided 
Assianment 5: Exploratory Essay 

(4-5 hours) 


9 


Read Wood, Ch. 8 
Proofs and fallacies 


Turn in Reading Journal 
Quiz 2 


10 


Read Wood pp. 339^48 on the writing process 
Read Wood, Ch. 13 


Assianment 6: Position Paper (7-8 hours) or Short 
Story Project (8-1 0 hours) 
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Week 


Reading/Discussion Topics 


Assignments & Assessments 


11 


Peer review workshop 
Read Wood, Ch. 9 


Turn in Reading Journal 
Quiz 3 


12 


Discuss MLA citation 


Assianment 7: Issue Research Project 

(2 hours) or Fiction Paper (3 hours) 


13 


Review Wood, Ch. 9 
Oral presentations 


Turn in Reading Journal 

Assessment 2; Oral Presentations Rubric provided 


14 


Argument Workshop 


Assianment 8: Documented Inquiry 

(5-6 hours) 


15 


Final Exam 



Sam p le Class Policies and Ex p ectations 

Students often enter college unaware of expectations regarding attendance, participation, 
behavior, grading, and academic integrity. Faculty frequently include detailed policies and 
expectations in their syllabi, making explicit to students the standards of successful 
participation. There are often institutional-level policies as well. 



Attendance Policy 

Regular attendance is required. School policy states that a student who is absent more 
than six hours of class may be viewed as administratively dropped from the course. 
Students who intend to withdraw from the course must do so by the official date or they 
will receive an “F” in the course. Please be aware that the state of Texas has begun to 
impose penalties on students who drop courses excessively. For example, if a student 
repeats the same course more than twice, that person will have to pay extra tuition. 
Beginning in Fall 2007, the Texas Legislature passed a law limiting students to no more 
than SIX total course withdrawals throughout their educational career in obtaining a 
certificate and/or degree. 

Consideration for excused absences is at the instructor’s discretion. Excused absences 
include official university activities, documented religious observance, illness with a 
physician’s note, or a verifiable death in the family. 

Expectations 

Student Conduct 

• Students are expected to attend class regularly and to complete reading and 
writing assignments on the day specified. 

• No food, drink, or tobacco of any kind is permitted in the classroom. 

• Cell phones and pagers must be turned off during class. 

• Disruptive behavior will not be tolerated. 
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• Examples of disruptive behavior include, but are not limited, to talking while the 
instructor is lecturing, discussing non-course related issues during class 
assignment time, and using profane language. 

Academic Behaviors 

• Students are expected to have adequate time management skills. 

• Students are expected to self-monitor their level of competence in the subject in 
order to be aware when improvement is needed. 

• Students are expected to seek out resources for improvement in understanding 
of the subject, such as study groups and tutoring. 

• Students are expected to practice efficient study and preparation skills outside of 
class. 

• Students are expected to take clear, concise class notes. 

• Students are expected to be able to collaborate and work in a team. 

• Students are expected to be able accept critical feedback including critiques of 
submitted work. 

Grading Policy 

Failure to turn in any essay will result in failure of this course! 

Grading. Scale 

• A (90-100) = excellent/performance beyond mastery 

• B (80-89) = above average/beyond basic mastery 

• C (70-79) = average mastery 

• D (60-69) = below average 

• F (0-59) = failure 
Evaluation Method 

• Essay #1 - 5% 

• Essay #2 - 1 0% 

• Essay #3 - 1 5% 

• Essay #4 (Research essay) - 25% 

• Oral Presentation of research - 5% 

• Final Exam - 20% 

• Daily Work - 10% 

• Quizzes - 1 0% 
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Sample ot a grading, rubric for essay, assignments! 

Grading Sheet: Total points possible = 100 

1 . Does the title tell the reader what the essay is about, and does the essay open 
with sufficient background information that sets a context for the topic and 
explains the controversy surrounding the topic? (0-10 points) 

2. Is the thesis clearly stated? (0-3 points) 

3. Do the examples and details in the essay clearly and sufficiently support the 
thesis? Does each paragraph relate to the thesis, and does each paragraph have 
a clear focus or topic sentence? (0-15 points) 

4. Does the writer acknowledge and refute opposing viewpoints? (o-io pts) 

5. Is the essay organized in an easy-to-follow and coherent fashion? (0-5 points) 

6. Does the writer provide transition words when needed between sentences, and 
does the writer provide transitions between paragraphs? (0-4 points) 

7. Is the wording clear, effective, and error free? Does the writer vary sentence 
structure, incorporate “vigorous” (active) verbs, and employ concrete, vivid 
language that evokes sensory images? Does the writer use a consistent point of 
view and verb tense? 

(0-10 points) 

8. Does the writer wrap up the essay with a concluding paragraph that effectively 
ties together the main points of the essay? (0-3 points) 

9. Does the writer use proper spelling, grammar, and punctuation? (0-40 points) 

Total Points 



Academic Integrity Policy 

Plagiarism will not be tolerated; be advised we use plagiarism detection software. The 
following, directly from the College Handbook, defines and sets the penalties for 
plagiarism and/or cheating: All academic work, written or otherwise, submitted by a 
student to an instructor or other academic supervisor, is expected to be the result of the 
student’s own thought, research, or self-expression. In cases in which students feel 
uncertain about a question of plagiarism, they are obliged to consult the instructor on 
the matter prior to submitting any work. When students submit work purported to be 
their own, but which in any way borrows ideas, organization, wording, or anything else 
from another source without appropriate acknowledgment, they are guilty of plagiarism. 
Plagiarism includes reproducing someone else’s work, whether it be from a published 
article, chapter of a book, or a paper from a friend. 

Plagiarism also includes the practice of employing or allowing another person to alter or 
revise the work submitted as one’s own. Students must acknowledge all outside 
sources of information. Making simple changes while leaving intact the organization and 
thoughts of others is plagiaristic. Plagiarism does not, however, extend to those ideas 
that are so generally and freely circulated as to be part of the public domain. Cheating 
includes, but is not limited to, students wrongfully giving, taking, or presenting any 
information or material with the intent of aiding themselves or others with any academic 
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work. Within seven working days of finding a student guilty of cheating or plagiarism, an 
instructor may choose to do one or more of the following: 

• Assign any grade to the paper or test involved and so notify the student in writing. 

• Assign any grade for the course in which the offending paper or examination was 
submitted and so notify the student in writing. 

• Recommend that the student be dropped from the course in which the offense 
occurred. Such a recommendation shall be made to the Division Chair. 

• Chair shall so recommend to the Dean of Instructional Services and send copies 
to the instructor and the student. 

• Recommend that the student be dropped from a program of a division. Such a 
recommendation shall be made to the Division Chair and to the Division Chair of 
the student’s program (if different); if agreement occurs, the Division Chair shall 
so recommend to the Dean of Instructional Services and send copies to the 
instructor and the student. 

• Recommend suspension or dismissal from the College. Such a recommendation 
shall be made to the Division Chair; if agreement occurs, the Division Chair shall 
so recommend to the Dean division and/or the chairperson of the division in 
which the student is enrolled in a program, to the instructor, and to the student. If 
the Dean concurs on a recommendation of suspension or expulsion from the 
College, the Dean shall take the necessary action. 

Sample List of Student Resources 

The following list is representative of the resources mentioned in 2008 entry-level college 
course syllabi in Texas. The resources listed are indicative of the expectation that post- 
secondary students take responsibility for their own learning. As often explicitly stated in 
course syllabi, students are expected to take advantage of these resources, and may even 
be required to do so. 



• Writing. Center 

The English department operates a free-of-charge Writing Center in building X. 
Students in English courses can receive in-depth help with writing problems from 
the Writing Center staff. Students who believe that they could benefit from 
concentrated, supervised work on some aspect of writing should attend the 
Writing Center more than once while working on their papers. 

• Student Success Programs 

The College supports a variety of programs to help connect students with the 
University and foster academic success. These programs include learning 
assistance, developmental education, advising and mentoring, admissions and 
transition, and federally-funded programs. Students requiring assistance 
academically, personally, or socially should contact the Office of Student 
Success Programs at xxx-xxx-xxxx for more information and appropriate 
referrals. 
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• Americans with Disabilities Act 

The college is on record as being committed to both the spirit and letter of 
federal equal opportunity legislation; reference Public Law 92-112 of The 
Rehabilitation Act of 1973 as amended. With the passage of federal legislation 
entitled Americans with Disabilities Act (ADA), pursuant to section 504 of the 
Rehabilitation Act, there is renewed focus on providing this population with the 
same opportunities enjoyed by all citizens. As a faculty member, instructors are 
required by law to provide "reasonable accommodations" to students with 
disabilities, so as not to discriminate on the basis of that disability. Student 
responsibility primarily rests with informing faculty of their need for 
accommodation and in providing authorized documentation through designated 
administrative channels. Information regarding specific diagnostic criteria and 
policies for obtaining academic accommodations can be found at 
www.abc.edu/disability. Also, students may visit the Office for Students with 
Disabilities in room X of building X or call (xxx) xxx-xxxx. 

• E-Culture Policy ' 

This course has adopted the College email address as an official means of 
communication with students. Through the use of email, the instructor is able to 
provide students with relevant and timely information, designed to facilitate 
student success. In particular, important information concerning registration, 
financial aid, payment of bills, and graduation may be sent to students through 
email. All students are assigned an email account and information about 
activating and using it is available at www.abc.edu. New students (first semester 
at the college) are able to activate their email account 24 hours after registering 
for courses. There is no additional charge to students for using this account, and 
it remains active as long as a student is enrolled at the college. Students are 
responsible for checking their email regularly. 



Su pp lementar y Documents 

The following list represents additional documents included within the Reference Course 
Profile, but not included in the Sample Schedule. These supplementary materials are 
intended to enhance both instruction and student learning, and may be used at instructors' 
discretion. 



• Revisions Handout 

• Using Quotations in a Literary Essay Handout 

• Works Cited Handout 

• Explanation of MLA Documentation Style Handout 

• Peer Response Questions 

• Bibliography Checklist 
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• Computer Lab Agenda 

• Direct Quotes Explanation 

• Scoring Rubric 



Improving Alignment Between Postsecondary and Secondary Education 



102 



epic 

I Educational Policy 
Improvement Center 



Appendix D: Scale Items & Rationale Statements 
For Alignment Between the CCRS and Entry-Level 
CTE Courses 

Most Necessary for Preparation to Succeed in this Course 

• This element is critical for success in the course 

• Course is taught with the assumption that students already know this information 

• This element will not be retaught in this course 

• Students will have difficulty succeeding in the course if they have not learned this 
element previously 

More Necessary for Preparation to Succeed in this Course 

• This element is important for success in the course 

• Course is taught with the assumption that students are at least familiar with or 
aware of this element 

• This element will be reviewed only and not retaught in this course 

• Students will benefit from having learned this element previous to the course but 
can probably relearn it during the course and still succeed in the course 

Less Necessary for Preparation to Succeed in this Course 

• Student knowledge of and familiarity with this element may be helpful 

• Course is taught with the assumption that students are familiar with this element 
on a very general level 

• This element will be taught in some detail in the class 

• Even if students have not learned this previously, they will be able to learn it in 
the course at a level sufficient to succeed in the course 

Least Necessary for Preparation to Succeed in this Course 

• Students need only minimal knowledge of and familiarity with this element 

• Course is taught with the assumption that students may be only vaguely aware of 
this element 

• This element will be taught as new material in this course even if students have 
been taught it before 

• Students will be able to succeed in this course even if they only have a very 
general awareness or understanding of this element when they enter the course 

Not Necessary for Preparation to Succeed in this Course 

• This element is too advanced for this course 

• This element will be encountered for the first time in subsequent courses in the 
subject area 

• This element is too specialized or specific for this course 

• This element is irrelevant to this course 
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• This element will be introduced as new material in this course with the 

assumption that students have not learned anything about it before this course 
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